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PREFACE to this edition 



Notes on Optimization was published in 1971 as part of the Van Nostrand Reinhold Notes on Sys- 
tem Sciences, edited by George L. Turin. Our aim was to publish short, accessible treatments of 
graduate-level material in inexpensive books (the price of a book in the series was about five dol- 
lars). The effort was successful for several years. Van Nostrand Reinhold was then purchased by a 
conglomerate which cancelled Notes on System Sciences because it was not sufficiently profitable. 
Books have since become expensive. However, the World Wide Web has again made it possible to 
publish cheaply. 

Notes on Optimization has been out of print for 20 years. However, several people have been 
using it as a text or as a reference in a course. They have urged me to re-publish it. The idea of 
making it freely available over the Web was attractive because it reaffirmed the original aim. The 
only obstacle was to retype the manuscript in LaTex. I thank Kate Klohe for doing just that. 

I would appreciate knowing if you find any mistakes in the book, or if you have suggestions for 
(small) changes that would improve it. 

Berkeley, California P.P. Varaiya 

September, 1998 
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PREFACE 



These Notes were developed for a ten-week course I have taught for the past three years to first-year 
graduate students of the University of California at Berkeley My objective has been to present, 
in a compact and unified manner, the main concepts and techniques of mathematical programming 
and optimal control to students having diverse technical backgrounds. A reasonable knowledge of 
advanced calculus (up to the Implicit Function Theorem), linear algebra (linear independence, basis, 
matrix inverse), and linear differential equations (transition matrix, adjoint solution) is sufficient for 
the reader to follow the Notes. 

The treatment of the topics presented here is deep. Although the coverage is not encyclopedic, 
an understanding of this material should enable the reader to follow much of the recent technical 
literature on nonlinear programming, (deterministic) optimal control, and mathematical economics. 
The examples and exercises given in the text form an integral part of the Notes and most readers will 
need to attend to them before continuing further. To facilitate the use of these Notes as a textbook, 
I have incurred the cost of some repetition in order to make almost all chapters self-contained. 
However, Chapter V must be read before Chapter VI, and Chapter VII before Chapter VIII. 

The selection of topics, as well as their presentation, has been influenced by many of my students 
and colleagues, who have read and criticized earlier drafts. I would especially like to acknowledge 
the help of Professors M. Athans, A. Cohen, C.A. Desoer, J-P. Jacob, E. Polak, and Mr. M. Ripper. I 
also want to thank Mrs. Billie Vrtiak for her marvelous typing in spite of starting from a not terribly 
legible handwritten manuscript. Finally, I want to thank Professor G.L. Turin for his encouraging 
and patient editorship. 

Berkeley, California P.P. Varaiya 

November, 1971 
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Chapter 1 



INTRODUCTION 



In this chapter, we present our model of the optimal decision-making problem, illustrate decision- 
making situations by a few examples, and briefly introduce two more general models which we 
cannot discuss further in these Notes. 

1.1 The Optimal Decision Problem 

These Notes show how to arrive at an optimal decision assuming that complete information is given. 
The phrase complete information is given means that the following requirements are met: 

1 . The set of all permissible decisions is known, and 

2. The cost of each decision is known. 

When these conditions are satisfied, the decisions can be ranked according to whether they incur 
greater or lesser cost. An optimal decision is then any decision which incurs the least cost among 
the set of permissible decisions. 

In order to model a decision-making situation in mathematical terms, certain further requirements 
must be satisfied, namely, 

1 . The set of all decisions can be adequately represented as a subset of a vector space with each 
vector representing a decision, and 

2. The cost corresponding to these decisions is given by a real-valued function. 
Some illustrations will help. 

Example 1: The Pot Company (Potco) manufacturers a smoking blend called Acapulco Gold. 
The blend is made up of tobacco and mary-john leaves. For legal reasons the fraction a of mary- 
john in the mixture must satisfy < a < ~. From extensive market research Potco has determined 
their expected volume of sales as a function of a and the selling price p. Furthermore, tobacco can 
be purchased at a fixed price, whereas the cost of mary-john is a function of the amount purchased. 
If Potco wants to maximize its profits, how much mary-john and tobacco should it purchase, and 
what price p should it set? 

Example 2: Tough University provides "quality" education to undergraduate and graduate stu- 
dents. In an agreement signed with Tough's undergraduates and graduates (TUGs), "quality" is 
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defined as follows: every year, each u (undergraduate) must take eight courses, one of which is a 
seminar and the rest of which are lecture courses, whereas each g (graduate) must take two seminars 
and five lecture courses. A seminar cannot have more than 20 students and a lecture course cannot 
have more than 40 students. The University has a faculty of 1000. The Weary Old Radicals (WORs) 
have a contract with the University which stipulates that every junior faculty member (there are 750 
of these) shall be required to teach six lecture courses and two seminars each year, whereas every 
senior faculty member (there are 250 of these) shall teach three lecture courses and three seminars 
each year. The Regents of Touch rate Tough's President at a points per u and j3 points per g "pro- 
cessed" by the University. Subject to the agreements with the TUGs and WORs how many u's and 
g's should the President admit to maximize his rating? 

Example 3: (See Figure 1.1.) An engineer is asked to construct a road (broken line) connection 
point a to point b. The current profile of the ground is given by the solid line. The only requirement 
is that the final road should not have a slope exceeding 0.001. If it costs $c per cubic foot to excavate 
or fill the ground, how should he design the road to meet the specifications at minimum cost? 

Example 4: Mr. Shell is the manager of an economy which produces one output, wine. There 
are two factors of production, capital and labor. If K(t) and L(t) respectively are the capital stock 
used and the labor employed at time t, then the rate of output of wine W(t) at time is given by the 
production function 



As Manager, Mr. Shell allocates some of the output rate W(t) to the consumption rate C(t), and 
the remainder I(t) to investment in capital goods. (Obviously, W, C, I, and K are being measured 
in a common currency.) Thus, W(t) = C{t) + I{t) = (1 - s{t))W{t) where s{t) = I{t)/W(t) 



G [0, 1] is the fraction of output which is saved and invested. Suppose that the capital stock decays 
exponentially with time at a rate 5 > 0, so that the net rate of growth of capital is given by the 
following equation: 



W(t) 



F(K(t),L(t)) 



a 




>• b 



Figure 1.1: Admissable set of example. 



K{t) 




(1.1) 



-SK(t) + s(t)W(t) 
-SK(t) + s(t)F(K(t),L(t)). 



The labor force is growing at a constant birth rate of /3 > 0. Hence, 
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L{t) 



f3L(t). 



(1.2) 



Suppose that the production function F exhibits constant returns to scale, i.e., F(XK,XL) = 
XF(K, L) for all A > 0. If we define the relevant variable in terms of per capita of labor, w = 
W/L,c= C/L,k = K/l, and if we let f(k) = F(k, I), then we see that F(K, L) — LF(K/L, 1) = 
Lf(k), whence the consumption per capita of labor becomes c{t) = (I — s(t))f(k(t)). Using these 
definitions and equations (1.1) and (1.2) it is easy to see that K{t) satisfies the differential equation 



where fi = (5 + (5). The first term of the right-hand side in (3) is the increase in the capital-to-labor 
ratio due to investment whereas the second terms is the decrease due to depreciation and increase in 
the labor force. 

Suppose there is a planning horizon time T, and at time Mr. Shell starts with capital-to-labor 
ratio k Q . If "welfare" over the planning period [0, T] is identified with total consumption f Q T c{t)dt, 
what should Mr. Shell's savings policy s(t), < t < T, be so as to maximize welfare? What 
savings policy maximizes welfare subject to the additional restriction that the capital-to-labor ratio 
at time T should be at least /c^? If future consumption is discounted at rate a > and if time horizon 
is oo, the welfare function becomes J °° e~ at c(t)dt. What is the optimum policy corresponding to 
this criterion? 

These examples illustrate the kinds of decision-making problems which can be formulated math- 
ematically so as to be amenable to solutions by the theory presented in these Notes. We must always 
remember that a mathematical formulation is inevitably an abstraction and the gain in precision may 
have occurred at a great loss of realism. For instance, Example 2 is caricature (see also a faintly re- 
lated but more more elaborate formulation in Bruno [1970]), whereas Example 4 is light-years away 
from reality. In the latter case, the value of the mathematical exercise is greater the more insensitive 
are the optimum savings policies with respect to the simplifying assumptions of the mathematical 
model. (In connection with this example and related models see the critique by Koopmans [1967].) 

In the examples above, the set of permissible decisions is represented by the set of all points 
in some vector space which satisfy certain constraints. Thus, in the first example, a permissible 
decision is any two-dimensional vector (a,p) satisfying the constraints < a < \ and < 
p. In the second example, any vector (u,g) with u > 0, g > 0, constrained by the number 
of faculty and the agreements with the TUGs and WORs is a permissible decision. In the last 
example, a permissible decision is any real-valued function s(t), < t < T, constrained by 
< s(t) < 1. (It is of mathematical but not conceptual interest to note that in this case a decision 
is represented by a vector in a function space which is infinite-dimensional.) More concisely then, 
these Notes are concerned with optimizing (i.e. maximizing or minimizing) a real-valued function 
over a vector space subject to constraints. The constraints themselves are presented in terms of 
functional inequalities or equalities. 



(1.3). 



k(t) 



s(t)f(k(t))-fik(t) 



(1.3) 
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At this point, it is important to realize that the distinction between the function which is to be 
optimized and the functions which describe the constraints, although convenient for presenting the 
mathematical theory, may be quite artificial in practice. For instance, suppose we have to choose 
the durations of various traffic lights in a section of a city so as to achieve optimum traffic flow. 
Let us suppose that we know the transportation needs of all the people in this section. Before we 
can begin to suggest a design, we need a criterion to determine what is meant by "optimum traffic 
flow." More abstractly, we need a criterion by which we can compare different decisions, which in 
this case are different patterns of traffic-light durations. One way of doing this is to assign as cost to 
each decision the total amount of time taken to make all the trips within this section. An alternative 
and equally plausible goal may be to minimize the maximum waiting time (that is the total time 
spent at stop lights) in each trip. Now it may happen that these two objective functions may be 
inconsistent in the sense that they may give rise to different orderings of the permissible decisions. 
Indeed, it may be the case that the optimum decision according to the first criterion may be lead to 
veiy long waiting times for a few trips, so that this decision is far from optimum according to the 
second criterion. We can then redefine the problem as minimizing the first cost function (total time 
for trips) subject to the constraint that the waiting time for any trip is less than some reasonable 
bound (say one minute). In this way, the second goal (minimum waiting time) has been modified 
and reintroduced as a constraint. This interchangeability of goal and constraints also appears at a 
deeper level in much of the mathematical theory. We will see that in most of the results the objective 
function and the functions describing the constraints are treated in the same manner. 

1.2 Some Other Models of Decision Problems 

Our model of a single decision-maker with complete information can be generalized along two 
very important directions. In the first place, the hypothesis of complete information can be relaxed 
by allowing that decision-making occurs in an uncertain environment. In the second place, we 
can replace the single decision-maker by a group of two or more agents whose collective decision 
determines the outcome. Since we cannot study these more general models in these Notes, we 
merely point out here some situations where such models arise naturally and give some references. 

1.2.1 Optimization under uncertainty. 

A person wants to invest $1,000 in the stock market. He wants to maximize his capital gains, and 
at the same time minimize the risk of losing his money. The two objectives are incompatible, since 
the stock which is likely to have higher gains is also likely to involve greater risk. The situation 
is different from our previous examples in that the outcome (future stock prices) is uncertain. It is 
customary to model this uncertainty stochastically. Thus, the investor may assign probability 0.5 to 
the event that the price of shares in Glamor company increases by $100, probability 0.25 that the 
price is unchanged, and probability 0.25 that it drops by $100. A similar model is made for all the 
other stocks that the investor is willing to consider, and a decision problem can be formulated as 
follows. How should $1,000 be invested so as to maximize the expected value of the capital gains 
subject to the constraint that the probability of losing more than $100 is less than 0.1? 

As another example, consider the design of a controller for a chemical process where the decision 
variable are temperature, input rates of various chemicals, etc. Usually there are impurities in the 
chemicals and disturbances in the heating process which may be regarded as additional inputs of a 
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random nature and modeled as stochastic processes. After this, just as in the case of the portfolio- 
selection problem, we can formulate a decision problem in such a way as to take into account these 
random disturbances. 

If the uncertainties are modelled stochastically as in the example above, then in many cases 
the techniques presented in these Notes can be usefully applied to the resulting optimal decision 
problem. To do justice to these decision-making situations, however, it is necessary to give great 
attention to the various ways in which the uncertainties can be modelled mathematically. We also 
need to worry about finding equivalent but simpler formulations. For instance, it is of great signif- 
icance to know that, given appropriate conditions, an optimal decision problem under uncertainty 
is equivalent to another optimal decision problem under complete information. (This result, known 
as the Certainty-Equivalence principle in economics has been extended and baptized the Separation 
Theorem in the control literature. See Wonham [1968].) Unfortunately, to be able to deal with 
these models, we need a good background in Statistics and Probability Theory besides the material 
presented in these Notes. We can only refer the reader to the extensive literature on Statistical De- 
cision Theory (Savage [1954], Blackwell and Girshick [1954]) and on Stochastic Optimal Control 
(Meditch [1969], Kushner [1971]). 

1.2.2 The case of more than one decision-maker. 

Agent Alpha is chasing agent Beta. The place is a large circular field. Alpha is driving a fast, heavy 
car which does not maneuver easily, whereas Beta is riding a motor scooter, slow but with good 
maneuverability. What should Alpha do to get as close to Beta as possible? What should Beta 
do to stay out of Alpha's reach? This situation is fundamentally different from those discussed so 
far. Here there are two decision-makers with opposing objectives. Each agent does not know what 
the other is planning to do, yet the effectiveness of his decision depends crucially upon the other's 
decision, so that optimality cannot be defined as we did earlier. We need a new concept of rational 
(optimal) decision-making. Situations such as these have been studied extensively and an elaborate 
structure, known as the Theory of Games, exists which describes and prescribes behavior in these 
situations. Although the practical impact of this theory is not great, it has proved to be among the 
most fruitful sources of unifying analytical concepts in the social sciences, notably economics and 
political science. The best single source for Game Theory is still Luce and Raiffa [1957], whereas 
the mathematical content of the theory is concisely displayed in Owen [1968]. The control theorist 
will probably be most interested in Isaacs [1965], and Blaquiere, et ai, [1969]. 

The difficulty caused by the lack of knowledge of the actions of the other decision-making agents 
arises even if all the agents have the same objective, since a particular decision taken by our agent 
may be better or worse than another decision depending upon the (unknown) decisions taken by the 
other agents. It is of crucial importance to invent schemes to coordinate the actions of the individual 
decision-makers in a consistent manner. Although problems involving many decision-makers are 
present in any system of large size, the number of results available is pitifully small. (See Mesarovic, 
et al, [1970] and Marschak and Radner [1971].) In the author's opinion, these problems represent 
one of the most important and challenging areas of research in decision theory. 
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Chapter 2 

OPTIMIZATION OVER AN OPEN 
SET 

In this chapter we study in detail the first example of Chapter 1 . We first establish some notation 
which will be in force throughout these Notes. Then we study our example. This will generalize 
to a canonical problem, the properties of whose solution are stated as a theorem. Some additional 
properties are mentioned in the last section. 

2.1 Notation 
2.1.1 

All vectors are column vectors, with two consistent exceptions mentioned in 2.1.3 and 2.1.5 below 
and some other minor and convenient exceptions in the text. Prime denotes transpose so that if 
x £ R n then x' is the row vector x' = (x±, . . . , x n ), and x = {x\, . . . , x n )' . Vectors are normally 
denoted by lower case letters, the ith component of a vector x £ R n is denoted Xi, and different 
vectors denoted by the same symbol are distinguished by superscripts as in x- 7 and x k . denotes 
both the zero vector and the real number zero, but no confusion will result. 

Thus if x = (x\, . . . , x n )' and y = (y%, . . . , y n )' then x'y = x\y\ + . . . + x n y n as in ordinary 
matrix multiplication. If x € R n we define \x\ = +Vx'x. 

2.1.2 

If x = (xi, . . . , x n )' and y = (yi, . . . , y n )' then x > y means Xj > yi, i = 1, . . . , n. In particular if 
x G R n , then x > 0, if X\ > 0, % = 1, . . . , n. 

2.1.3 

Matrices are normally denoted by capital letters. If A is an m x n matrix, then A? denotes the jth 
column of A, and Ai denotes the ith row of A. Note that A4 is a row vector. A\ denotes the entry 
of A in the ith row and jth column; this entry is sometimes also denoted by the lower case letter 
dij, and then we also write A = {ciij}. I denotes the identity matrix; its size will be clear from the 
context. If confusion is likely, we write I n to denote the n x n identity matrix. 
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2.1.4 

If / : R n — ► R m is a function, its ith component is written fa, i = 1, . . . , m. Note that : i? n — > 
Sometimes we describe a function by specifying a rule to calculate f(x) for every x. In this case 
we write / f(x). For example, if A is an m x n matrix, we can write F : x \—> Ax to denote 

the function / : R n — ► R m whose value at a point x G R n is Ax. 



2.1.5 

If / : R n i — i R is a differentiable function, the derivative of / at x is the row vector ((df /dxi)(x), . . . , (df/dx n )(x)). 
This derivative is denoted by (df/dx)(x) or or df /dx\ x=x or / x | x =x, and if the argument x 
is clear from the context it may be dropped. The column vector (f x (x))' is also denoted V x f(x), 
and is called the gradient of / at x. If / : (x, y) i— > /(a;, y) is a differentiable function from 
i? n x i? m into i?, the partial derivative of / with respect to x at the point (x, y) is the n-dimensional 
row vector f x (x,y) = (df/dx)(x,y) = ((df/dx{)(x, y), (df/dx n )(x, y)), and similarly 
= (df/dy)(x,y) = ((df/d yi )(x, y), . . . , (df/dy m )(x, y)). Finally, if / : R n -+ R m is 
a differentiable function with components fi, . . . , f m , then its derivative at x is the m x n matrix 



Tx {x) = f x x 



flx(x) 
fmx{x) 

dx! \ x ) 



dx„ V X J 



9fm 
8x n 



(x) 



2.1.6 

If / : R n — > R is twice differentiable, its second derivative at x is the nxn matrix (d 2 f / dxdx) (x) = 
fxx(x) where {f X x{x)){ = (d 2 f /dxjdxi)(x). Thus, in terms of the notation in Section 2.1.5 above, 

fxx(x) = (d/dx)(f x Y(x). 



2.2 Example 

We consider in detail the first example of Chapter 1. Define the following variables and functions: 



a = fraction of mary-john in proposed mixture, 

p = sale price per pound of mixture, 

v = total amount of mixture produced, 

f(a,p) = expected sales volume (as determined by market research) of mixture as a function of (a, p) . 
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Since it is not profitable to produce more than can be sold we must have: 

v = f(a,p), 

m = amount (in pounds) of mary-john purchased, and 
t = amount (in pounds) of tobacco purchased. 



Evidently, 

m = av, and 
t = (I — a)v. 

Let 

Pi(m) = purchase price of m pounds of mary-john, and 
P2 = purchase price per pound of tobacco. 

Then the total cost as a function of a, p is 

C(a,p) = P 1 (af(a,p)) + P 2 (l - a)f(a,p). 

The revenue is 

R(a,p) =pf(a,p), 



so that the net profit is 

N(a,p) = R(a,p) — C(a,p). 

The set of admissible decisions is 17, where = {(a,p)|0 < a < ^,0 < p < 00}. Formally, we 
have the the following decision problem: 

Maximize N(a,p), 
subject to (a,p) G £1. 

Suppose that (a*,p*) is an optimal decision, i.e., 

(a*,p*)en and 

N(a*,p*)> N(a,p) for all (a,p) G ft. K '> 

We are going to establish some properties of (a*,p*). First of all we note that fl is an open subset 
of R 2 . Hence there exits e > such that 

(a,p) G fl whenever \(a,p) — (a*,p*)\ < e (2.2) 

In turn (2.2) implies that for every vector h = (/ii,^)' i n R 2 there exists n > (77 of course 
depends on h) such that 

((a*,p*) + 5{h 1 ,h 2 )) G n for < 5 < n (2.3) 
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a*,p*)+5(h 1 ,h 2 ) 




Figure 2.1: Admissable set of example. 



N(a* + 6hx,p* + 5h 2 ) = +5[^(S*,p*)h 1 + ™-( a *,p*)h 2 ] 



Combining (2.3) with (2.1) we obtain (2.4): 

N(a*,p*)>N(a* + Sh 1 ,p*+Sh 2 ) for < 5 < rj 
Now we assume that the function N is differentiable so that by Taylor's theorem 

N(a*,p*) 

where 

f -> as 5^0. 

Substitution of (2.5) into (2.4) yields 

cW 

Dividing by 5 > gives 



> <y[^(a*,p*)7n + ^(a*,P*> 2 ] + o(5). 



0>[^(a*, P *)h 1 + ^(a\p*)h 2 } + °-&. 
Letting 5 approach zero in (2.7), and using (2.6) we get 

> + f (a*,p*)h 2 ]. 



(2.4) 



(2.5) 



(2.6) 



(2.7) 



(2.8) 



Thus, using the facts that N is differentiable, (a* ,p*) is optimal, and 5 is open, we have concluded 
that the inequality (2.9) holds for every vector h <E R 2 . Clearly this is possible only if 



= 0, m a *, P *) = o. 



(2.9) 



Before evaluating the usefulness of property (2.8), let us prove a direct generalization. 
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2.3 The Main Result and its Consequences 

2.3.1 Theorem 

Let O be an open subset of R n . Let /: R n — > R be a differentiable function. Let x* be an optimal 
solution of the following decision-making problem: 

Maximize f{x) 
subject to x G Q. 

Then 

1(0 = 0. (2.11) 
Proof: Since x* G 17 and $7 is open, there exists e > such that 

x G f2 whenever |x — x*| < e. (2.12) 
In turn, (2.12) implies that for every vector h G i? n there exits 77 > (?? depending on h) such that 

(x* + 5h) G $7 whenever < 5 < ??. (2.13) 
Since x* is optimal, we must then have 

f(x*) > f(x* + 5h) whenever < 5 < r). (2.14) 
Since / is differentiable, by Taylor's theorem we have 

f{x* + Sh) = f(x*) + %(x*)5h + o(5), (2.15) 

where 

^1^0 as 5^0 (2.16) 
Substitution of (2.15) into (2.14) yields 

0>5%(x*)h + o(8) 

and dividing by 5 > gives 

>§(**)/* (2-17) 
Letting 5 approach zero in (2.17) and taking (2.16) into account, we see that 

0>%(x*)h, (2.18) 
Since the inequality (2.18) must hold for every h G R n , we must have 

= §£(*•). 

and the theorem is proved. 
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Table 2.1 




Does there exist 


At how many points 






an optimal deci- 


in is 2.2.2 


Further 


Case 


sion for 2.2.1? 


satisfied? 


Consequences 


1 


Yes 


Exactly one point, 


x* is the 






say x* 


unique optimal 


2 


Yes 


More than one point 




3 


No 


None 




4 


No 


Exactly one point 




5 


No 


More than one point 





2.3.2 Consequences. 

Let us evaluate the usefulness of (2.11) and its special case (2.18). Equation (2.11) gives us n 
equations which must be satisfied at any optimal decision x* = (x\, . . . , a;* )'. 
These are 

&GO = °, &0O = 0,..., = (2-19) 

Thus, every optimal decision must be a solution of these n simultaneous equations of n variables, so 
that the search for an optimal decision from is reduced to searching among the solutions of (2.19). 
In practice this may be a very difficult problem since these may be nonlinear equations and it may 
be necessary to use a digital computer. However, in these Notes we shall not be overly concerned 
with numerical solution techniques (but see 2.4.6 below). 

The theorem may also have conceptual significance. We return to the example and recall the 
N = R — C. Suppose that R and C are differentiable, in which case (2.18) implies that at every 
optimal decision (a* ,p*) 

or, in the language of economic analysis, marginal revenue = marginal cost. We have obtained an 
important economic insight. 

2.4 Remarks and Extensions 

2.4.1 A warning. 

Equation (2. 1 1) is only a necessary condition for x* to be optimal. There may exist decisions x G $7 
such that f x (x) = but x is not optimal. More generally, any one of the five cases in Table 2. 1 may 
occur. The diagrams in Figure 2.1 illustrate these cases. In each case = (—1, 1). 

Note that in the last three figures there is no optimal decision since the limit points -1 and +1 are 
not in the set of permissible decisions £1 = (—1,1). In summary, the theorem does not give us any 
clues concerning the existence of an optimal decision, and it does not give us sufficient conditions 
either. 
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-1 Case 1 1 -1 Case 2 1 -1 Case 3 i 




Case 4 Case 5 



Figure 2.2: Illustration of 4.1. 

2.4.2 Existence. 

If the set of permissible decisions is a closed and bounded subset of R n , and if / is continuous, 
then it follows by the Weierstrass Theorem that there exists an optimal decision. But if f2 is closed 
we cannot assert that the derivative of / vanishes at the optimum. Indeed, in the third figure above, 
if Q = [— 1, 1], then +1 is the optimal decision but the derivative is positive at that point. 

2.4.3 Local optimum. 

We say that x* E $7 is a locally optimal decision if there exists e > such that f(x*) > f(x) 
whenever x G f! and \x* — x\ < e. It is easy to see that the theorem holds (i.e., 2.11) for local 
optima also. 

2.4.4 Second-order conditions. 

Suppose / is twice-differentiable and let x* G $7 be optimal or even locally optimal. Then f x (x*) = 
0, and by Taylor's theorem 

f(x* + Sh) = f(x*) + \S 2 h! f xx (x*)h + o(5 2 ), (2.20) 

where ^p^- — > as S — > 0. Now for 8 > sufficiently small f(x* + Sh) < f(x*), so that dividing 
by 5 2 > yields 

0> ltif xx (x*)h + ?p 

and letting 5 approach zero we conclude that h'f xx (x*)h < for all h G R n . This means that 
f xx (x*) is a negative semi-definite matrix. Thus, if we have a twice differentiable objective function, 
we get an additional necessary condition. 

2.4.5 Sufficiency for local optimal. 

Suppose at x* G O, f x (x*) = and f xx is strictly negative definite. But then from the expansion 
(2.20) we can conclude that x* is a local optimum. 
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2.4.6 A numerical procedure. 

At any point x G f2 the gradient \/ x f(x) is a direction along which f(x) increases, i.e., f(x + s \/ x 
f(x)) > f(x) for all e > sufficiently small. This observation suggests the following scheme for 
finding a point x* G O which satisfies 2.1 1. We can formalize the scheme as an algorithm. 

Step 1. Pick x° G n. Set i = 0. Go to Step 2. 
Step 2. Calculate \j x f{x i ). If Vi/(^) = 0, stop. 

Otherwise let = x % + y z /(V) and go 

to Step 3. 

Step 3. Set i = i + 1 and return to Step 2. 

The step size di can be selected in many ways. For instance, one choice is to take di to be an 
optimal decision for the following problem: 

Max{/(x* + d v. f(x*))\d > 0, {x i +dy x fix')) G U}. 

This requires a one-dimensional search. Another choice is to let di = if f(x l + <ij_i \/ x 
f(x 1 )) > f(x l ); otherwise let di = 1/k where k is the smallest positive integer such that 
f(x l + 1/k cfj_i Vx f{x 1 )) > f{x % ). To start the process we let cLi > be arbitrary. 

Exercise: Let / be continuous differentiable. Let {di} be produced by either of these choices and 
let 

{xi} be the resulting sequence. Then 

1. f(x i+ i) > f(xi) ifx i+1 / Xi,i 

2. if x* G fi is a limit point of the sequence {xi}, f x (x*) = 0. 

For other numerical procedures the reader is referred to Zangwill [1969] or Polak [1971]. 



Chapter 3 



OPTIMIZATION OVER SETS 
DEFINED BY EQUALITY 
CONSTRAINTS 

We first study a simple example and examine the properties of an optimal decision. This will 
generalize to a canonical problem, and the properties of its optimal decisions are stated in the form 
of a theorem. Additional properties are summarized in Section 3 and a numerical scheme is applied 
to determine the optimal design of resistive networks. 

3.1 Example 

We want to find the rectangle of maximum area inscribed in an ellipse denned by 

h(x,y) = £ + £ = a. (3.1) 
The problem can be formalized as follows (see Figure 3.1): 

Maximize fo(x,y) = Axy 

subject to (x, y) eft = {(x, y)\fi(x, y) = a}. 

The main difference between problem (3.2) and the decisions studied in the last chapter is that 
the set of permissible decisions ft is not an open set. Hence, if (x* ,y*) is an optimal decision we 
cannot assert that fo(x*,y*) > fo(x, y) for all (x, y) in an open set containing (x*,y*). Returning 
to problem (3.2), suppose (x*,y*) is an optimal decision. Clearly then either x* / or y* ^ 0. Let 
us suppose y* ^ 0. Then from figure 3.1 it is evident that there exist (i)e > 0, (ii) an open set V 
containing (x*,y*), and (iii) a differentiable function g : (x* — e, x* + e) — > V such that 

fi(x,y) = a and (x,y) eV iff fy = g{x). 1 (3.3) 

In particular this implies that y* = g(x*), and that fi(x,g(x)) = a whenever \x — x*\ < e. Since 

'Note that y* 7^ implies f\ v (x*, Y*) 7^ 0, so that this assertion follows from the Implicit Function Theorem. The 
assertion is false if y* = 0. In the present case let < s < a — x* and g{x) = +b[a — (x/a) 2 ] 1 ^ 2 . 
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Tangent plane to 

U at (x*,y*) 




Figure 3.1: Illustration of example. 



(x*,y*) = (x*,g(x*)) is optimum for (3.2), it follows that x* is an optimal solution for (3.4): 

Maximize f (x) = f (x,g(x)) 
subject to \x — x*\ < e. 



(3.4) 



But the constraint set in (3.4) is an open set (in R 1 ) and the objective function fo is differentiable, 
so that by Theorem 2.3.1, fo x (x*) = 0, which we can also express as 



hx{x*,y*) + fo y (x*,y*)g x {x*) = 
Using the fact that fi(x, g(x)) = a for \x — x* \ <e, we see that 

fix(x*,y*) + fi y (x*,y*)g x (x*) = 0, 
and since fi y (x*,y*) / Owe can evaluate g x (x*), 

g x (x*) = -f{yfix(x*,y*), 
and substitute in (3.5) to obtain the condition (3.6): 

fox ~ foyfiyfix = at (x*,y*). 



(3.5) 



(3.6) 



Thus an optimal decision (x*,y*) must satisfy the two equations fi(x*,y*) = a and (3.6). Solving 
these yields 

x* = t{a/2) l ' 2 a , y* = t{a/2) l l 2 b. 



3.2. GENERAL CASE 
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Evidently there are two optimal decisions, (x* ,y*) = l(a/2) 1 / 2 (a, b), and the maximum area is 

m(a) = 2aab. (3.7) 

The condition (3.6) can be interpreted differently. Define 

y = fo y fiy(x*,y*). (3.8) 

Then (3.6) and (3.8) can be rewritten as (3.9): 

(/o*,/otf) = A*(/i x ,/i„) at (x*,y*) (3.9) 

In terms of the gradients of /o, /i, (3.9) is equivalent to 

V f (x*,y*) = [ V f 1 (x*,y*)}\*, (3.10) 

which means that at an optimal decision the gradient of the objective function /q is normal to the 
plane tangent to the constraint set fl 
Finally we note that 

A* = fg. (3.11) 

where m(a) = maximum area. 

3.2 General Case 

3.2.1 Theorem. 

Let fi : R n — > R, i = 0, 1, . . . , m (m < n), be continuously differentiable functions and let x* be 
an optimal decision of problem (3.12): 

Maximize f (x) 

subject to fi{x) = a>i, i = 1, . . . , m. 

Suppose that at x* the derivatives fi x {x*),i = 1, . . . , m, are linearly independent. Then there exists 
a vector A* = (A*, . . . , A^)' such that 

fox(x*) = X*Ji x (x*) + ... + X* m fm X (x*) (3.13) 

Furthermore, let m{a\ , . . . , a m ) be the maximum value of (3. 12) as a function of a = (cti , . . . , a m )' . 
Let x*(a) be an optimal decision for (3.12). If x*(a) is a differentiable function of a then m(a) is 
a differentiable function of a, and 

(A*)' = If (3.14) 

Proof. Since fi x {x*),i = 1, . . . , m, are linearly independent, then by re-labeling the coordinates of 
x if necessary, we can assume that the mxm matrix [(dfi/dxj)(x*)] , 1 < i,j < m, is nonsingular. 
By the Implicit Function Theorem (see Fleming [1965]) it follows that there exist (i) e > 0, (ii) an 
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open set V in R n containing x* , and (iii) a differentiable function g : U — > R m , where U = 

[(x m+1 , . . .,x n )}\ \x m+ i - x* m+e \ < e,£ = 1, . . . ,n -m], such that 

fi(x±,. . . ,x n ) = cti,l < i < m, and {x\, . . . , x n ) £ V 

iff 

),l<j<m, and (x m+x , . . . , x n ) G U (3.15) 

(see Figure 3.2). 

In particular this implies that x* = 9j{x* m+l , . . . , x* ), 1 < j < rn, and 

fi{g{x m +i, ■ ■ ■ ,x n ),x m+ i,. . . ,x n ) = an , i = l,...,m. (3.16) 

For convenience, let us define w = (xi, . . . , x m )' , u = (x m+ i, . . . , x n )' and / = (/i, . . . , f m )'. 
Then, since x* = (w*,u*) = (g(u*),u*) is optimal for (3.12), it follows that u* is an optimal 
decision for (3.17): 

Maximize f (u) = f (g(u),u) 
subject to u G U. 

But U is an open subset of R n ~ m and /o is a differentiable function on U (since /o and g are 
differentiable), so that by Theorem 2.3.1 , fo u (u*) = 0, which we can also express using the chain 
rule for derivatives as 

hu{u*) = fow(x*)g u (u*) + f 0u (x*) = 0. (3.18) 
Differentiating (3.16) with respect to u = (x m+ i, . . . , x n )' , we see that 

fw(x*)9u(u*) + fu(x*) = 0, 
and since the m x m matrix f w (x*) is nonsingular we can evaluate g u {u*), 

9u(u*) = -[f w (x*)]^f u (x*), 
and substitute in (3.18) to obtain the condition 

-fowtfU + fou = at x* = (w*,u*). (3.19) 
Next, define the m-dimensional column vector A* by 

(A*)' = fowf^\x*. (3.20) 
Then (3.19) and (3.20) can be written as (3.21): 

ifow(x*), fou(x*)) = (X*Y(fw(x*),f u (x*))- (3.21) 
Since x = (w,u), this is the same as 

f0x(x*) = (\*)'fx(x*) = XtfUx*) + ... + \* m fmx(x*), 
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X\ , . . . , x m 




Figure 3.2: Illustration of theorem. 

which is equation (3.13). 

To prove (3.14), we vary a in a neighborhood of a fixed value, say a. We define w*(a) = 
(x*(a), . . . ,xj^(a))' and u*(a) = (x^ n+l (a), . . . ,x*a))'. By hypothesis, f w is nonsingular at 
x*(a). Since f(x) and x*(a) are continuously differentiable by hypothesis, it follows that f w is 
nonsingular at x* (a) in a neighborhood of a, say N. We have the equation 

f(w*(a),u*(a)) = a, (3.22) 

-fo w f^fu + fou = at K(a),u*(a)), (3.23) 
for a£JV. Also, m(a) = /o(x*(a)), so that 

= fo w w* a + fouK (3.24) 
Differentiating (3.22) with respect to a gives 

UK + f u u* a = J, 

so that 

w a ~l" /to fu u a /u, , 
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and multiplying on the left by fo w gives 

Using (3.23), this equation can be rewritten as 

h w w* a + f 0u u* a = fou,/' 1 . (3.25) 
In (3.25), if we substitute from (3.20) and (3.24), we obtain (3.14) and the theorem is proved. 

3.2.2 Geometric interpretation. 

The equality constraints of the problem in 3.12 define an — m dimensional surface 

fl = {x\fi{x) = a i} i = 1, . . . , m}. 

The hypothesis of linear independence of {fi x (x*)\l < i < m} guarantees that the tangent plane 
through 12 at x* is described by 

{h\f ix (x*)h = , i=l,...,m}, (3.26) 

so that the set of (column vectors orthogonal to this tangent surface is 

{Ai Vx fi(x*) + . . . + A m Vx fm{x*)\Xi G R,i = l,...,m}. 

Condition (3.13) is therefore equivalent to saying that at an optimal decision x* , the gradient of the 
objective function \/ x fo( x *) is normal to the tangent surface (3.12). 

3.2.3 Algebraic interpretation. 

Let us again define w = (x\, . . . , x m )' and u = (x m+ i, . . . , x n )' . Suppose that f w (x) is nonsin- 
gular at some point x = (w, u) in 12 which is not necessarily optimal. Then the Implicit Function 
Theorem enables us to solve, in a neighborhood of x, the m equations f(w, u) = a. u can then vary 
arbitrarily in a neighborhood of u. As u varies, w must change according to w = g(u) (in order to 
maintain f(w, u) = a), and the objective function changes according to fo(u) = fo(g(u),u). The 
derivative of /o at u is 

fou(u) = fo w g u + foux = -X'f u (x) + f Qu (x), 

where 

A' = /o™/-i, (3-27) 

Therefore, the direction of steepest increase of /o at u is 

Vuh{u) = -f u {x)\ + f 0u {x) , (3.28) 

and if u is optimal, \/ u fo{u) = which, together with (3.27) is equation (3.13). We shall use (3.27) 
and (3.28) in the last section. 
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3.3 Remarks and Extensions 

3.3.1 The condition of linear independence. 

The necessary condition (3.13) need not hold if the derivatives fi x (x*), 1 < i < m, are not linearly 
independent. This can be checked in the following example 



Minimize 

subject to sin(xf + xj) (3.29) 
%{x\ + x'q) = 1. 



3.3.2 An alternative condition. 

Keeping the notation of Theorem 3.2.1, define the Lagrangian function L : R n + m _> R by L : 
(x, A) i — y fo(x) — YliLi \fi{ x )- The following is a reformulation of 3.12, and its proof is left as 
an exercise. 

Let x* be optimal for (3.12), and suppose that fi x (x*), 1 < i < m, are linearly independent. 
Then there exists A* € R m such that (x*, A*) is a stationary point of L, i.e., L x {x* , A*) = and 
L x (x*,X*) = 0. 



3.3.3 Second-order conditions. 

Since we can convert the problem (3.12) into a problem of maximizing /o over an open set, all 
the comments of Section 2.4 will apply to the function /q. However, it is useful to translate these 
remarks in terms of the original function /o and /. This is possible because the function g is 
uniquely specified by (3.16) in a neighborhood of x*. Furthermore, if / is twice differentiable, so 
is g (see Fleming [1965]). It follows that if the functions /j, < i < m, are twice continuously 
differentiable, then so is /o, and a necessary condition for x* to be optimal for (3.12) and (3. 13) and 
the condition that the (n — m) x (n — m) matrix fo uu {u*) is negative semi-definite. Furthermore, 
if this matrix is negative definite then x* is a local optimum, the following exercise expresses 
ffouu{u*) in terms of derivatives of the functions f\. 



Exercise: Show that 



huu{u*) = [g' u ll] 



L 



fju 

I 



(w*,u* 



where 

in 

1=1 
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3.3.4 A numerical procedure. 

We assume that the derivatives fi x {x), 1 < i < m, are linearly independent for all x. Then the 
following algorithm is a straightforward adaptation of the procedure in Section 2.4.6. 

Step 1. Find xq arbitrary so that fi(x°) = ot{, 1 < i < m. Set k = and go to Step 2. 
Step 2. Find a partition x = (w, u) 2 of the variables such that f w (x k ) is nonsingular. Calculate A fc 
by (A fc )' = fowf-^y and y/ fc (u fc ) = -f u {x k )\ k + f^ u (x k ). If yf k (u k ) = 0, stop. Otherwise 
go to Step 3. 

Step 3. Set u k = u k + (4 y fo( uk )- Find sucn that fi(w k ,u k ) = 0, 1 < i < m. Set 
x fc+i _ (yj k : u k ), set fc = + 1, and return to Step 2. 

Remarks. As before, the step sizes c4 > can be selected various ways. The practical applicability 
of the algorithm depends upon two crucial factors: the ease with which we can find a partition 
x = (w, u) so that f w (x k ) is nonsingular, thus enabling us to calculate X k ; and the ease with which 
we can find w k so that f(w k , u k ) = a. In the next section we apply this algorithm to a practical 
problem where these two steps can be carried out without too much difficulty. 

3.3.5 Design of resistive networks. 

Consider a network N with n + 1 nodes and b branches. We choose one of the nodes as datum 
and denote by e = (ei, . . . , e n )' the vector of node-to-datum voltages. Orient the network graph 
and let v = (v±, . . . , and j = (ji, . . . ,jb)' respectively, denote the vectors of branch voltages 
and branch currents. Let A be the n x b reduced incidence matrix of the network graph. Then the 
Kirchhoff current and voltage laws respectively yield the equations 

Aj = and A'e = v (3.30) 

Next we suppose that each branch k contains a (possibly nonlinear)resistive element with the form 
shown in Figure 3.3, so that 

3k ~ jsk = 9k(v r k) = 9k{vk ~ v sk ), 1 <k <b, (3.31) 

where v r k is the voltage across the resistor. Here j s k, v s k are the source current and voltage in the 
kth branch, and is the characteristic of the resistor. Using the obvious vector notation j s € R b , 
v s G R b for the sources, v r G R b for the resistor voltages, and g = {g\, . . . , <#,)' ', we can rewrite 
(3.30) as (3.31): 

j ~ 3s = g(v - v s ) = g(v r ). (3.32) 

Although (3.30) implies that the current (jk—j s k) through the kth resistor depends only on the 
voltage v r k = {vk—Vsk) across itself, no essential simplification is achieved. Hence, in (3.31) we 
shall assume that gj. is a function of v r . This allows us to include coupled resistors and voltage- 
controlled current sources. Furthermore, let us suppose that there are I design parameters p = 
(pi, . . . ,pi)' which are under our control, so that (3.31) is replaced by (3.32): 

3 ~ jx = g{v r ,p) = g{v-v s ,p). (3.33) 



2 This is just a notational convenience. The w variable may consist of any m components of x. 
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Figure 3.3: The kth branch. 



If we combine (3.29) and (3.32) we obtain (3.33): 

Ag(A'e-v s ,p) 



(3.34) 



where we have denned i s = Aj s . The network design problem can then be stated as finding p, v s , i s 
so as to minimize some specified function /o(e,p, v s ,i s ). Formally, we have the optimization prob- 
lem (3.34): 



(3.35) 



Minimize f {e,p,v s ,i s ) 
subject to Ag(A'e — v s ,p) — i s = 0. 

We shall apply the algorithm 3.3.4 to this problem. To do this we make the following assumption. 

Assumption: (a) /o is differentiable. (b) g is differentiable and the nx n matrix A(dg / dv){v,p)A' 
is nonsingular for all v G R b ,p G Br. (c) The network N described by (3.33) is determinate i.e., 
for every value of (p, v s ,i s ) there is a unique e = E(p, v s ,i s ) satisfying (3.33). 

In terms of the notation of 3.3.4, if we let x = (e,p,v s ,i s ), then assumption (b) allows us to 
identify w = e, and u = (p, v s ,i s ). Also let f(x) = f(e,p, v s ,i s ) = Ag(A'e—v s ,p) — i s . Now the 
crucial part in the algorithm is to obtain \ k at some point x k . To this end let x, = (e,p, v s ,i s ) be a 
fixed point. Then the corresponding A = A is given by (see (3.27)) 

A' = fow(x)fvHx) = f 0e (x)f-Hx). (3.36) 
From the definition of / we have 

f e (x) = AG(v r ,p)A', 

where v r = A'e — v s , and G(v r ,p) = (dg/dv r )(v r ,p). Therefore, A is the solution (unique by 
assumption (b)) of the following linear equation: 



AG'(v r ,p)A'\ = f[ )e (x) 



(3.37) 



Now (3.36) has the following extremely interesting physical interpretation. If we compare (3.33) 
with (3.36) we see immediately that A is the node-to-datum response voltages of a linear network 
N(v r ,p) driven by the current sources /g e (x). Furthermore, this network has the same graph as 
the original network (since they have the same incidence matrix); moreover, its branch admittance 
matrix, G'(v r ,p), is the transpose of the incremental branch admittance matrix (evaluated at (v r ,p)) 
of the original network N. For this reason, N(v r ,p) is called the adjoint network (of N) at (v r ,p). 
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Once we have obtained A we can obtain y u /o(n) using (3.28). Elementary calculations yield 
(3.37): 



(3.38) 



/$p( fi ) 




" [g(fir,fOM # " 




" /o P (^) 








A + 








—I 







Vu/o(«) 

We can now state the algorithm. 

(p°, «°, i°) arbitrary. Solve (3.33) to obtain e° = E(p°,°' { 



o^,ig). Let 



Oand 



Step 1. Select u° 
go to Step 2. 

Step 2. Calculate v k = A'e k — v k . calculate fQ e (x k ). Calculate the node-to-datum response A fc of 
the adjoint network N(v k ,p k ) driven by the current source fQ e (x k ). Calculate \/ u fo{u k ) from 
(3.37). If this gradient is zero, stop. Otherwise go to Step 3. 

dk Vm fo(u ), where df. > is a predetermined 



Step 3. Letu k+1 = (p k+1 ,v k+1 ,i k+1 ) 
step size. 3 Solve (3.33) to obtain e k+1 



{Ep 



fc+i 



-1 ,-fcH 



1 ). Set = A; + 1 and return to Step 2. 



Remark 1. Each iteration from u k to requires one linear network analysis step (the 
computation of \ k in Step 2), and one nonlinear network analysis step (the computation of e k+1 in 
step 3). This latter step may be very complex. 

Remark 2. In practice we can control only some of the components of v s and i s , the rest being 
fixed. The only change this requires in the algorithm is that in Step 3 we set 



P 



fc+i 



P — dkfb p {u ) just as before, where as v 



k+i 



v S j ~ dk{dfo/dv sj )(u k ) and 



•fc+i 



\ m ~ dk{dfo/di sm )(u ) with j and m ranging only over the controllable components and 
the rest of the components equal to their specified values. 

Remark 3. The interpretation of A as the response of the adjoint network has been exploited for 
particular function /q in a series of papers (director and Rohrer [1969a], [1969b], [1969c]). Their 
derivation of the adjoint network does not appear as transparent as the one given here. Although 
we have used the incidence matrix A to obtain our network equation (3.33), one can use a more 
general cutset matrix. Similarly, more general representations of the resistive elements may be 
employed. In every case the "adjoint" network arises from a network interpretation of (3.27), 

[fw{x)}'\ = fow(x), 

with the transpose of the matrix giving rise to the adjective "adjoint." 

Exercise: [DC biasing of transistor circuits (see Dowell and Rohrer [1971]).] Let N be a transistor 
circuit, and let (3.33) model the dc behavior of this circuit. Suppose that i s is fixed, v s j for j € J 
are variable, and v s j for j ^ J are fixed. For each choice of v s j , j G J, we obtain the vector e and 
hence the branch voltage vector v = A'e. Some of the components vt, t G T, will correspond to 
bias voltages for the transistors in the network, and we wish to choose v s j , j G J, so that v t is as 
close as possible to a desired bias voltage vf, t G T. If we choose nonnegative numbers at, with 
relative magnitudes reflecting the importance of the different transistors then we can formulate the 
criterion 



3 Note the minus sign in the expression u k — y„ fo(u k ). Remember we are minimizing fo, which is equivalent to 
maximizing (—fo)- 
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/o(e) = ^a t \v t -vi\ 2 . 
teT 

(i) Specialize the algorithm above for this particular case. 

(ii) How do the formulas change if the network equations are written using an arbitrary cutset matrix 
instead of the incidence matrix? 
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Chapter 4 



OPTIMIZATION OVER SETS 
DEFINED BY INEQUALITY 
CONSTRAINTS: LINEAR 
PROGRAMMING 

In the first section we study in detail Example 2 of Chapter I, and then we define the general linear 
programming problem. In the second section we present the duality theory for linear program- 
ming and use it to obtain some sensitivity results. In Section 3 we present the Simplex algorithm 
which is the main procedure used to solve linear programming problems. In section 4 we apply 
the results of Sections 2 and 3 to study the linear programming theory of competitive economy. 
Additional miscellaneous comments are collected in the last section. For a detailed and readily ac- 
cessible treatment of the material presented in this chapter see the companion volume in this Series 
(Sakarovitch [1971]). 

4.1 The Linear Programming Problem 
4.1.1 Example. 

Recall Example 2 of Chapter I. Let g and u respectively be the number of graduate and undergradu- 
ate students admitted. Then the number of seminars demanded per year is 29 ^ u , and the number of 
lecture courses demanded per year is 59 ^J U ■ On the supply side of our accounting, the faculty can 
offer 2(750) + 3(250) = 2250 seminars and 6(750) + 3(250) = 5250 lecture courses. Because of 
his contractual agreements, the President must satisfy 

^ < 2250 or 2g + u < 45, 000 

and 

^±£i < 5250 or 5g + 7u < 210, 000 . 
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Since negative g or u is meaningless, there are also the constraints g > 0, u > 0. Formally then the 
President faces the following decision problem: 

Maximize ag + (3u 
subject to 2g + u< 45, 000 

5g + 7u < 210,000 ( ' } 

g > 0,u > . 

It is convenient to use a more general notation. So let a; = (g,u)', c= (a, /?)', b = (45000,210000,0,0)' 
and let A be the 4x2 matrix 



,4 



2 1 

5 7 

-1 

-1 



Then (44) can be rewritten as (4.2)' 



Maximize c'x 

(4 2) 

subject to Ax < 6 . 



Let Ai, 1 < i < 4, denote the rows of 4. Then the set f2 of all vectors x which satisfy the constraints 
in (4.2) is given by = {x\ AiX < 6j, 1 < i < 4} and is the polygon OPQR in Figure 4.1. 

For each choice x, the President receives the payoff c'x. Therefore, the surface of constant payoff 
k say, is the hyperplane ir(k) = {x\c'x = k}. These hyperplanes for different values of k are 
parallel to one another since they have the same normal c. Furthermore, as k increases ir(k) moves 
in the direction c. (Obviously we are assuming in this discussion that c / 0.) Evidently an optimal 
decision is any point x* G f2 which lies on a hyperplane 7r(fc) which is farthest along the direction 
c. We can rephrase this by saying that x* G 17 is an optimal decision if and only if the plane tt* 
through x* does not intersect the interior of $7, and futhermore at x* the direction c points away 
from $7. From this condition we can immediately draw two very important conclusions: (i) at least 
one of the vertices of O is an optimal decision, and (ii) x* yields a higher payoff than all points 
in the cone K* consisting of all rays starting at x* and passing through since K* lies "below" 
7r*. The first conclusion is the foundation of the powerful Simplex algorithm which we present in 
Section 3. Here we pursue consequences of the second conclusion. For the situation depicted in 
Figure 4.1 we can see that x* = Q is an optimal decision and the cone K* is shown in Figure 4.2. 
Now x* satisfies A x x* = bi, A 2 x* = 6 2 , and A 3 x* < b 3 , A^x* < 64, so that K* is given by 

K* = {x* + h\A x h < , A 2 h < 0} . 



Since c'x* > c'y for all y G K* we conclude that 

c'h < for all h such that A x h < 0, A 2 h < . (4.3) 
We pause to formulate the generalization of (4.3) as an exercise. 



Recall the notation introduced in 1.1.2, so that x < y means Xi < yi for all i. 
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Figure 4.1: SI = OPQi?. 

Exercise 1: Let Aj, 1 < i < A;, be n-dimensional row vectors. Let c £ and let 6j, 1 < i < k, 
be real numbers. Consider the problem 



Maximize c'x 

subject to AiX < bi, 1 < i < k . 

For any x satisfying the constraints, let I{x) C {1, . . . , n} be such that Ai{x) = bi,i £ /(x), AjX < 
bi,i ^ -f(x). Suppose x* satisfies the constraints. Show that x* is optimal if an only if 



c'h < for all h such that Aih <0, iG I(x*). 

Returning to our problem, it is clear that (4.3) is satisfied as long as c lies between A\ and A 2 . 

Mathematically this means that (4.3) is satisfied if and only if there exist > 0, A2 > such that 

2 

J = \* 1 ,A 1 + \* 2 A 2 . (4.4) 

As c varies, the optimal decision will change. We can see from our analysis that the situation is as 
follows (see Figure 4.1): 



2 Although this statement is intuitively obvious, its generalization to n dimensions is a deep theorem known as Farkas' 
lemma (see Section 2). 
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Figure 4.2: K* is the cone generated by f2 at x*. 



1. x * = Q is optimal iff c lies between A\ and ^4 2 iff d = \\A\ + A|A 2 for some A* > 0, A 2 > 
0, 

2. x* G QP is optimal iff c lies along A 2 iff c' = X 2 A 2 for some A 2 > 0, 

3. x* = P is optimal iff c lies between A 3 and A 2 iff d = A^ 2 + \\A 3 for some \\ > 0, A3 > 
0, etc. 

These statements can be made in a more elegant way as follows: 

x* G O is optimal iff there exists A* > , 1 < i < 4, such that 

4 

(a) d = J2 x *ai , (b) if A* x* < V then A* = . (4.5) 

i=l 

For puiposes of application it is useful to separate those constraints which are of the form X{ > 0, 
from the rest, and to reformulate (4.5) accordingly We leave this as an exercise. 

Exercise 2: Show that (4.5) is equivalent to (4.6), below. (Here Ai = (an, ai 2 ).) x* G is optimal 
iff there exist A^ > , A 2 > such that 

(a) a < X*au + A 2 a 2i , i = 1, 2, 

(b) if ajix\ + a j2 x 2 < bj then x* = 0, j = 1,2. (4.6) 

(c) if Cj < A*j + A^a 2i then x* = 0, i = 1, 2. 
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4.1.2 Problem formulation. 

A linear programming problem (or LP in brief) is any decision problem of the form 4.7. 

Maximize c\X\ + c^x^ + ■ ■ ■ + c n x n 
subject to 

auxi + a i2 x 2 + • • • + a in x n < bi , I < i < k , 

auX! + + a in x n > bi , k + 1 <i<£ , 

auxi + + a in x n = bi , £ + 1 < i < m , 



and 



xj > , 1 < j < p , 

xj > , p + 1 < j < q; (4.7) 
xj arbitary , q + 1 < j < n , 



where the fixed real numbers. 

There are two important special cases: 
Case I: (4.7) is of the form (4.8): 



Maximize CjXj 



3=1 



subject to a ij x j ^ b i > 1 <i<m 
3=1 

Xj > , 1 < j ; < n 



(4.8) 



(4.9) 



Ca5e //: (4.7) is of the form (4.9): 



Maximize CjXj 
n 

subject to ^ aijXj = 6i , 1 < i < m , 
3=1 

Xj > , 1 < j ; < n . 
Although (4.7) appears to be more general than (4.8) and (4.9), such is not the case. 



Proposition: Every LP of the form (4.7) can be transformed into an equivalent LP of the form (4.8). 
Proof. 

Step 1: Replace each inequality constraint ^ o-ijXj > h by J2(~ a ij) x j — ( — 
Step 2: Replace each equality constraint Yl a ij x j = h by two inequality constraints: 

Y^aijXj < h, Y.(- a ij) x j < (-h)- 

Step 3: Replace each variable Xj which is constrained Xj < by a variable yj = — Xj constrained 
yj > and then replace aijXj by (-aij)yj for every i and CjXj by (-Cj)yj. 
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Step 4: Replace each variable xj which is not constrained in sign by a pair of variables 

Vj—Zj = Xj constrained y 3 > 0, Zj > and then replace aijXj by a^yj + (—Q>ij)zj for every i and 

CjXj by CjUj + (-Cj)zj. Evidently the resulting LP has the form (4.8) and is equivalent to the 

original one. 

Proposition: Eveiy LP of the form (4.7) can be transformed into an equivalent LP of the from (4.9) 

Proof. 

Step 1: Replace each inequality constraint ^ aijXj < h by the equality constraint 

2~2 ciijXj + yi = b{ where yi is an additional variable constrained yi > 0. 

Step 2: Replace each inequality constraint a ij x j > h by the equality constraint 

QijXj — yi = h where yi is an additional variable constrained by yi > 0. (The new variables 
added in these steps are called slack variables.) 

Step 3, Step 4: Repeat these steps from the previous proposition. Evidently the new LP has the 
form (4.9) and is equivalent to the original one. 



4.2 Qualitative Theory of Linear Programming 
4.2.1 Main results. 

We begin by quoting a fundamental result. For a proof the reader is referred to (Mangasarian 
[1969]). 

Farkas' Lemma. Let Ai, 1 < i < k, be n-dimensional row vectors. Let c G R n be a column vector. 
The following statements are equivalent: 

(i) for all x G R n , AiX < for 1 < i < k implies c'x < 0, 

k 

(ii) there exists Ai > 0, . . . , > such that d = KAi. 

i=l 

An algebraic version of this result is sometimes more convenient. 
Farkas' Lemma (algebraic version). Let Abe a k x n matrix. Let c S R n . The following statements 
are equivalent. 

(i) for all x G R n , Ax < implies c'x < 0, 

(ii) there exists A > 0, A G R k , such that A' A = c. 

Using this result it is possible to derive the main results following the intuitive reasoning of (4.1). 
We leave this development as two exercises and follow a more elegant but less intuitive approach. 

Exercise 1: With the same hypothesis and notation of Exercise 1 in 4. 1, use the first version of 
Farkas' lemma to show that there exist A* > for i G I(x*) such that A* Ai = c . 

iei(x*) 

Exercise 2: Let x* satisfy the constraints for problem (4.17). Use the previous exercise to show 
that x* is optimal iff there exist A* > 0, . . . , A^ > such that 

m 

(a) Cj < ^2 K a ij > 1 < 3 < n 

i=i 

n m 

(b) if ^ aijX* < bi then A* = , 1 < i < m (c) if ^ K a ij > c j tnen x* = , 1 < j < m. 

j=i i=i 

In the remaining discussion, c G R n , b G n are fixed vectors, and A = {a,ij} is a fixed m x n 
matrix, whereas x G R n and A G R m will be variable. Consider the pair of LPs (4.10) and (4.11) 
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below. (4.10) is called the primal problem and (4.11) is called the dual problem. 



Maximize c\X\ + . . . + c n x n 

subject to a>nx\ + . . . +ai n x n < bi , 1 < i < m 

Xj > , 1 < j < n . 



Maximize 
subject to 



-I- . . . + X m b m 
\\a\j + . . . + A 



in u "mj ^ Cj , 

A* > , 



1 < i < m . 



(4.11) 



Definition: Let = {x G i? n | Ac < 6, x > 0} be the set of all points satisfying the constraints 
of the primal problem. Similarly let ^ = {A e R m \X'A > d, X > 0}. A point x G J7 P (A G $1^) is 
said to be a feasible solution ox feasible decision for the primal (dual). 

The next result is trivial. 
Lemma 1: (Weak duality) Let x G U p , A G f^. Then 



c'x < X' Ax < X'b. 



(4.12) 



Proof: x > and A'^4 — c' > implies (A'i-c'jx > giving the first inequality. b—Ax > and 
A' > implies X'(b— Ax) > giving the second inequality. 
Corollary 1: If x* G O and A* G such that c'x* = (X*)'b, then x* is optimal for (4.10) and A* is 
optimal for (4.11). 

Theorem 1: (Strong duality) Suppose f2 p ^ <fi and / ^. Then there exists x* which is optimum 
for (4.10) and A* which is optimum for (4.11). Furthermore, c'x* = (X*)'b. 

Proof: Because of the Corollary 1 it is enough to prove the last statement, i.e., we must show that 
there exist x > 0, A > 0, such that Ax < b, A'X > c and b'X—c'x < 0. By introducing slack 
variables y G R m , [i G R m , r G R, this is equivalent to the existence ofx > 0,y > 0,A > 5/ u < 
0, r < such that 



' A 














A' 


-In 




. -d 




b' 




1 _ 



X 



By the algebraic version of Farkas' Lemma, this is possible only if 

A'i - c9 < ,£<(), 
Aw = bO < , -w < , 
5 < 

implies 



(4.13) 



&'£ + c'tt? < 0. 



(4.14) 
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Case (i): Suppose (w, £, 9) satisfies (4.13) and 9 < 0. Then G U d , (w/-9) G U p , so that by 
Lemma 1 c'w/(—9) < b'£/9, which is equivalent to (4.14) since 9 < 0. 

Case (ii): Suppose (w, f , 9) satisfies (4.13) and 9 = 0, so that -A'£ > 0, -f > 0, Aw < 0, to > 0. 
By hypothesis, there exist x G Op, A G fi<j. Hence, = &'(-£) > = > 0, 

and c'w < (A'X)'w = X'(Aw) < 0. So that &'£ + c'w < 0. 

The existence part of the above result can be strengthened. 

Theorem 2: (i) Suppose £l p / <fi. Then there exists an optimum decision for the primal LP iff 

(ii) Suppose Q d 4>- Then there exists an optimum decision for the dual LP iff £l p ^ <fi. 
Proof Because of the symmetry of the primal and dual it is enough to prove only (i). The 
sufficiency part of (i) follows from Theorem 1, so that only the necessity remains. Suppose, in 
contradiction, that = <fi. We will show that sup {c'x\x G O p } = +oo. Now, £l d = $ means 
there does not exist A > such that A'X > c. Equivalently, there does not exist A > 0, fi < such 
that 



= [c] 

By Fai - kas' Lemma there exists w G R n such that Aw < 0, — w < 0, and c'w > 0. By hypothesis, 
Op 7^ 4>, so there exists x > such that Ax < b. but then for any 9 > 0, A(x + 9w) < b, 
(x + 9w) > 0, so that (x + 9w) G O p . Also, c'(x + 9w) = c'x + 9c'w. Evidently then, sup 
{c'xjx G Op} = +oo so that there is no optimal decision for the primal. 

Remark: In Theorem 2(i), the hypothesis that £l p ^ cj) is essential. Consider the following exercise. 
Exercise 3: Exhibit a pair of primal and dual problems such that neither has a feasible solution. 
Theorem 3: (Optimality condition) x* G Q p is optimal if and only if there exists A* G Q d such that 



A' 



OijXj < bi implies A* = , 

3=1 

and (4.15) 

m 

X*aij < Cj implies x* = . 

i=l 

((4.15) is known as the condition of complementary slackness.) 

Proof: First of all we note that for x* G fl p , X* G fl d , (4.15) is equivalent to (4.16): 

(X*y(Ax* - b) = 0, and (A'X* - c)'x* = . (4.16) 

Necessity. Suppose x* G £l p is optimal. Then from Theorem 2, £l d / <fi, so that by Theorem 1 
there exists A* G Q d such that c'x* = (X*)'b. By Lemma 1 we always have 
c'x* < (X*)'Ax* < (X*)'b so that we must have c'x* = (X*)'Ax* = (X*)'b. But (4.16) is just an 
equivalent rearrangement of these two equalities. 

Sufficiency. Suppose (4.16) holds for some x* G £l p , X* G Q d . The first equality in (4.16) yields 
(\*)'b = (A*)'^x* = (A'X*)'x*, while the second yields (A'X*)'x* = c'x*, so that c'x* = (X*)'b. 
By Corollary 1, x* is optimal. 
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The conditions x* G O p , x* G Qd hi Theorem 3 can be replaced by the weaker x* > 0, A* > 
provided we strengthen (4.15) as in the following result, whose proof is left as an exercise. 

Theorem 4: (Saddle point) x* > is optimal for the primal if and only if there exists A* > such 
that 

L(x,X*) < L(x*,X*) < L(x*,X) for all x > 0, and allA > 0, (4.17) 
where L: R n xR m — > R is defined by 

L(x, A) = c'x - X'{Ax - b) (4.18) 

Exercise 4: Prove Theorem 4. 

Remark. The function L is called the Lagrangian. A pair (x*, A*) satisfying (4.17) is said to form 
a saddle-point of L over the set {x\x G R n , x > 0} x {A| A G A > 0}. 

4.2.2 Results for problem (4.9). 

It is possible to derive analogous results for LPs of the form (4.9). We state these results as exercises, 
indicating how to use the results already obtained. We begin with a pair of LPs: 

Maximize c\X\ + . . . + c n x n 

subject to aux\ + . . . + ai n x n = h , 1 < i < m , (4.19) 

x j > , 1 < j < n . 

Minimize Ai&i + . . . + X m b m ^ 
subject to Aiaij + . . . + X m a m j > Cj , 1 < j < n . 

Note that in (4.20) the Aj are unrestricted in sign. Again (4. 19) is called the primal and (4.20) the 
dual. We let Q p , 17^ denote the set of all x, X satisfying the constraints of (4.19), (4.20) respectively. 

Exercise 5: Prove Theorems 1 and 2 with Q p and 0^ interpreted as above. (Hint. Replace (4.19) 
by the equivalent LP: maximize c'x, subject to Ax < b, (— A)x < (—6), x > 0. This is now of the 
form (4.10). Apply Theorems 1 and 2.) 

Exercise 6: Show that x* G Q p is optimal iff there exists A* G suc h that 

m 

x* > implies ^ K a ij = c j ■ 
i=i 

Exercise 7: x* > is optimal iff there exists A* G R m such that 

L(x, X*) < L(x*,X*) < L(x*,X) for all x > 0, A G R rn . 

where L is defined in (4.18). (Note that, unlike (4.17), A is not restricted in sign.) 
Exercise 8: Formulate a dual for (4.7), and obtain the result analogous to Exercise 5. 
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4.2.3 Sensitivity analysis. 

We investigate how the maximum value of (4.10) or (4.19) changes as the vectors b and c change. 
The matrix A will remain fixed. Let O p and 0^ be the sets of feasible solutions for the pair (4.10) and 
(4. 1 1) or for the pair (4. 19) and (4.20). We write tt p (b) and £ld(c) to denote the explicit dependence 
on b and c respectively. Let B = {b G R m \n p (b) / <j>} and C = {c G # n |O d (c) / (/>}, and for 
(i),c)£BxC define 

M(b,c) = max {c'x|x G O p (fe)} = min {A'fr|A G n d (c)} . (4.21) 

For 1 < i < m, £ G i?, 6 G i? m denote 

= (h,b 2 , . . . ,bi-i,bi +e,b i+1 , . . .,b m )' , 

and for 1 < j < n, e G R, c G R n denote 

c(j,e) = (ci,c 2 , • • . ,Cj-i,Cj + e,Cj + i,. . .,c n )' . 

We define in the usual way the right and left hand partial derivatives of M at a point (b, c) G B x C 
as follows: 



36, 



(b,c) 



lim 
£ > 



1 -{M(b(i,e),c)-M(b,c)} , 



dM- 



(&,c) = lim ±{M(6,c) 
e -> 
e > 



M(b(i, -e),c)} , 



dcj 



(b,c)= lim i{M(S,c(j,e))-M(M} 

£ 
£ > 



lim ±{M(S, 5 -M(6,c(i, -£))}, 

£ -» 
£ > 

o o 

Let B, C denote the interiors of B, C respectively. 

o o 

Theorem 5: At each (b, c) EB x (7, the partial derivatives given above exist. Furthermore, if 
x G A G 0(2 (c) are optimal, then 

^(M)< A* < 2£?(b,c), l<i<m, (4.22) 
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^(M)>%>^f (S,c), l<j<n, (4.23) 

Proof: We first show (4.22), (4.23) assuming that the partial derivatives exist. By strong duality 
M(b, c) = X'b, and by weak duality M (b(i, e), c) < X'b(i, e), so that 

l{M(b(i,e),c) - M(b,c)} < \\'{b{i, e) - b}X h for e > 0, 
i{M(S,c) -M(S(i, -£),£)} > ±A'{6-6(i,-e)} = A;, fore > 0. 

Taking limits as e — > 0, e > 0, gives (4.22). 

On the other hand, M (b , c) = c'x, and M(6, c(j, e)) > (c(j, e))'x, so that 

±{M(S, c(i, e)) - M(b, c)} > i{c(i, e)' - c}'x = x j} for e > 0, 
i{M(6, c) - M(t>, c(j, -e))} < \{c - c(j, -e)}'x = xj, for e > 0, 

which give (4.23) as e — > 0, e > 0. 

Finally, the existence of the right and left partial derivatives follows from Exercises 8, 9 below. 
We recall some fundamental definitions from convex analysis. 

Definition: X C R n is said to be convex if x, y € X and < 6 < 1 implies (#x + (1 — 9)y) G X. 

Definition: Let X C i? n and / : X — ► R. (i) / is said to be convex if X is convex, and x,y £ X, 
< 6 < 1 implies /(6»x + (1 - 0)y) < 9f(x) + (1 - 0)/(y). (ii) / is said to be concave if -/ is 
convex, i.e., x, y G X, < 6» < 1 implies /(0x + (1 - 0)y) > 6f(x) + (1 - 6>)/(y). 

Exercise 8: (a) Show that O p , Q^, and the sets i? C R rn , C C i? n defined above are convex sets, 
(b) Show that for fixed c £ C, M(-,c) : B ^ Ris concave and for fixed b € B, M(6, •) : C i? 
is convex. 

Exercise 9: Let X C and / : X — > R be convex. Show that at each point x in the interior of 
X, the left and right hand partial derivatives of / exist. (Hint: First show that for 

e 2 >e 1 >0>5 1 > 5 2 ,(l/e 2 ){f(x(i,e 2 )) - /(£)} > (l/ei){/(x(i J ei)) - /(£))} > 

> (l/5 2 ){/(^(«,^2))-/(a)}. Then the result follows immediately.) 
Remark 1: Clearly if (dM/dbi)(b) exists, then we have equality in (4.22), and then this result 
compares with 3.14). 

Remark 2: We can also show without difficulty that M(-, c) and M (b, •) are piecewise linear (more 
accurately, linear plus constant) functions on B and C respectively. This is useful in some 
computational problems. 

Remark 3: The variables of the dual problem are called Lagrange variables or dual variables or 
shadow-prices. The reason behind the last name will be clear in Section 4. 



4.3 The Simplex Algorithm 
4.3.1 Preliminaries 

We now present the celebrated Simplex algorithm for finding an optimum solution to any LP of the 
form (4.24): 

Maximize c\X\+ ... + c n x n 

subject to aux\ + . . . + a,i n x n = bi , 1 < i < m (4.24) 

xj > , 1 < j < n . 
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As mentioned in 4. 1 the algorithm rests upon the observations that if an optimal exists, then at least 
one vertex of the feasible set $7 P is an optimal solution. Since f2 p has only finitely many vertices (see 
Corollary 1 below), we only have to investigate a finite set. The practicability of this investigation 
depends on the ease with which we can characterize the vertices of $7 p . This is done in Lemma 1. 

In the following we let A? denote the jth column of A, i.e., A? = (aij, • • • , a m j)' '• We begin with 
a precise definition of a vertex. 

Definition: x G S7 P is said to be a vertex of £l p if x = Xy + (1 — X)z, with y, z in O p and 

< A < 1, implies x = y = z. 

Definition: For x G U p , let I(x) = {j\xj > 0}. 

Lemma 1: Let x G U p . Then x is a vertex of S7 P iff {A^\j G I(x)} is a linearly independent set. 
Exercise 1: Prove Lemma 1. 

m | 

Corollary 1: £l v has at most -. -r vertices. 

Lemma 2: Let x* be an optimal decision of (4.24). Then there is a vertex z* of £l p which is optimal. 
Proof: If {Ai\j G /(a;*)} * s linearly independent, let z* = x* and we are done. Hence suppose 
{A 3 \j G is linearly dependent so that there exist jj, not all zero, such that 

V jjAi = . 



For 9 £ R define z(0) G i? n by 

Zj{0) 



x) = Q lj , jel(x*) 

x*=0 , j£I(x*). 



Az(9) = Y, *#M i = £ x)A' + Y 7jA j 
j&i(x*) jei(x») jei{x*) 

= 6 + 0-0 = 6. 

Since x* > for j G I(x*), it follows that z(0) > when 

|0| < min { ^| | j G J(x*) } = 0* say . 
Hence z(0) G O p whenever \9\ < 9*. Since x* is optimal we must have 

c'x* > dz{9) = c'x* + 9 Y c iVi for -*0<9 <9* . 

jei{x*) 

Since 9 can take on positive and negative values, the inequality above can hold on if £ Cj^/j = 

J£l(x*) 

0, and then c'x* = c'z(9), so that z{9) is also an optimal solution for \9\ < 9*. But from the 
definition of z{9) it is easy to see that we can pick 9q with \9q \ = 9* such that zj(9q ) = x*+0o7j = 
for at least one j = jo in I(x*). Then, 



I(z(9 )) C I(x*) - {j } 
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Again, if {A 3 \j G I{z{0q))} is linearly independent, then we let z* = z{0q) and we are done. 
Otherwise we repeat the procedure above with z(9q). Clearly, in a finite number of steps we will 
find an optimal decision z* which is also vertex. 

At this point we abandon the geometric term "vertex" and how to established LP terminology. 

Definition: (i) z is said to be a basic feasible solution if z G Cl p , and G I(z)} is linearly 

independent. The set I(z) is then called the basis at z, and Xj, j G I(z), are called the basic 
variables at z. Xj, j G" I{z) are called the non-basic variables at z. 

Definition: A basic feasible solution z is said to be non-degenerate if I{z) has m elements. 

Notation: Let z be a non-degenerate basic feasible solution, and let j\ < j2 < • • • < jm 

constitute I{z). Let D(z) denote the m x m non-singular matrix D(z) = [A 51 :^ 2 : . . . :A J ' m ], let 
c(z) denote the m-dimensional column vector c(z) = (c^, . . . ,Cj m )' and define X(z) by A'(z) = 
c'(z)[.D(z)] _1 . We call A(z) the shadow-price vector at z. 
Lemma 3: Let z be a non-degenerate basic feasible solution. Then z is optimal if and only if 

\'{z)A > Cj , for all , j I(z) . (4.25) 

Proof: By Exercise 6 of Section 2.2, z is optimal iff there exists A such that 

X'Ai = Cj , for , j G I(z) , (4.26) 



A' A 7 > ,for , j J(z) , (4.27) 
But since z is non-degenerate, (4.26) holds iff A = A(z) and then (4.27) is the same as (4.25). 

4.3.2 The Simplex Algorithm. 

The algorithm is divided into two parts: In Phase I we determine if f2 p is empty or not, and if not, 
we obtain a basic feasible solution. Phase II starts with a basic feasible solution and determines if 
it is optimal or not, and if not obtains another basic feasible solution with a higher value. Iterating 
on this procedure, in a finite number of steps, either we obtain an optimum solution or we discover 
that no optimum exists, i.e., sup {c'x|x G = +oo. We shall discuss Phase II first. 
We make the following simplifying assumption. We will comment on it later. 

Assumption of non-degeneracy. Every basic feasible solution is non-degenerate. 
Phase II: 

Step 1. Let z° be a basic feasible solution obtained from Phase I or by any other means. Set k = 
and go to Step 2. 

Step 2. Calculate [D(z k )]~ 1 ,c(z k ), and the shadow-price vector \'(z k ) = c'(z k )[D(z k )]~ 1 . For 
each j I(z k ) calculate Cj — \'{z k )AK If all these numbers are < 0, stop, because z k is optimal 
by Lemma 3. Otherwise pick any j G" I{z k ) such that c- — X'(z k )A^ > and go to Step 3. 
Step 3. Let I(z k ) consist of Ji < J2 < • • • < jm- Compute the vector 

l k = (t^j i • • • JjmY = [D{z k )]~ 1 Ai. If j k < 0, stop, because by Lemma 4 below, there is no 
finite optimum. Otherwise go to Step 4. 

Step 4. Compute 9 = min {{z k ^ k )\j G i{z),j k > 0}. Evidently < 9 < oo. Define z k+1 by 
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z j ~ ®7j 




e 


' 3 ' = I 


4 = 


i 3+3 



4 +1 = < , j=j (4.28) 



By Lemma 5 below, z k+1 is a basic feasible solution with c'z k+1 > c'z k . Set k = k + 1 and return 
to Step 2. 

Lemma 4: If j k < 0, sup {c'^jx G f2 p } = oo. 
Proof: Define z(9) by 



' Zj - erf , j g /(z) 

, J=J ^ (4.29) 

k zj = , j ^0) and j / j . 



First of all, since 7 fc < it follows that z{6) > for 9 > 0. Next, Az(0) = Az - ^ 7* A- 3 ' + 
9A j = Az by definition of 7 fc . Hence, z(9) G ST2 p for > 0. Finally, 



~'z(0) = c ' 2 ~ 0c'(z k )j k + 0c~. 



= c'z + 9{c- - c'(z k )[D(z k )]~ 1 A^} (4.30) 
= c'z + 0{c-. - A'(> fe )A^}i . 

But from step 2 {cj - A'(z fc ) A'} > 0, so that c'z(9) -> 00 as -> 00. 
Lemma 5: z k+l is a basic feasible solution and c'z fc+1 > c'z fc . 

Prao/- Let j G I(z k ) be such that 7~ fc > and z| = 9jk. Then from (4.28) we see that z k+1 = 0, 
hence 



I(z k+1 )c(I(z)-{j})U{j} , (4.31) 

so that it is enough to prove that A is independent of {A|j G I(z), j + j}. But if this is not the 
case, we must have 7- = 0, giving a contradiction. Finally if we compare (4.28) and (4.29), we see 
from (4.30) that 

dz k + 1 -dz k = 9{c- j -i{z k )Ai} , 
which is positive from Step 2. 

Corollary 2: In a finite number of steps Phase II will obtain an optimal solution or will determine 
that supjc'xjx G f2 p } = 00. 

Corollary 3: Suppose Phase II terminates at an optimal basic feasible solution z*. Then 7(2*) is an 
optimal solution of the dual of (4.24). 
Exercise 2: Prove Corollaries 2 and 3. 

Remark 1: By the non-degeneracy assumption, I(z k+l ) has m elements, so that in (4.31) we must 
have equality. We see then that D(z k+1 ) is obtained from D(z k ) by replacing the column A by 
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the column AK More precisely if D(z k ) = [A jl : . . . \A j '- 1 ':A j: .A j '+ 1 \ . . . :A jm ] and if 

jk < 3 < jk+i then D(z k+1 ) = [A^\ . . . :A^- 1 ':A^+ 1 ': . . . \A^\A^.A^+^. . . . \A^\. Let E be the 

matrix E = [A^ 1 : . . . •A> i - 1 :A>:A> i +i': . . . Then [D^ 1 ))' 1 = P E~ l where the matrix P 



permutes the columns of D(z k+l ) such that E 
to check that E^ 1 = MfD^)]" 1 where 



D(z k+1 )P. Next, if A> = ^ Jj e A je , it is easy 



M 



T 

ith column 

Then [D(z k+1 )}^ 1 = PM[D(z k )}~\ so that these inverses can be easily computed. 

Remark 2: The similarity between Step 2 of Phase II and Step 2 of the algorithm in 3.3.4 is 
striking. The basic variables at z k correspond to the variables w k and non4jasic variables 
correspond to u k . For each j I(z k ) we can interpret the number cj — X'(z k )Aj to be the net 
increase in the objective value per unit increase in the jth component of z k . This net increase is due 
to the direct increase Cj minus the indirect decrease X'(z k )Aj due to the compensating changes in 
the basic variables necessary to maintain feasibility. The analogous quantity in 3.3.4 is 

(df /d Uj )(x k ) - (\ k y(df/du,)(x k ). 

Remark 3: By eliminating any dependent equations in (4.24) we can guarantee that the matrix A 
has rank n. Hence at any degenerate basic feasible solution z k we can always find I(z k ) D I(z k ) 
such that I{z k ) has m elements and {Aj\j G I(z k )} is a linearly independent set. We can apply 
Phase II using I(z k ) instead of I(z k ). But then in Step 4 it may turn out that 9 = so that 
z k+i _ £ fc reason f or t hj s j s that I(z k ) is not unique, so that we have to try various 
alternatives for I(z k ) until we find one for which 9 > 0. In this way the non-degeneracy 
assumption can be eliminated. For details see (Canon, et al, [1970]). 

We now describe how to obtain an initial basic feasible solution. 

Phase I: 

Step I. by multiplying some of the equality constraints in (4.24) by — 1 if necessary, we can assume 
that b > 0. Replace the LP (4.24) by the LP (4.32) involving the variables x and y: 



in 



Maximize — > % 

y (4.32) 

subject to aux\ + . . . + a in x n + = bi , 1 < i < m , 
Xj > , fji > , l<j<n,l<i<m. 
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Go to step 2. 

Step 2. Note that (x°, y°) = (0, b) is a basic feasible solution of (4.32). Apply phase II to (4.32) 
starting with this solution. Phase II must terminate in an optimum based feasible solution (x* , y* ) 



since the value of the objective function in (4.32) lies between — > bi and 0. Go to Step 3. 



Step 3. If y* = 0, x* is a basic feasible solution for (4.24). If y* / 0, by Exercise 3 below, (4.24) 
has no feasible solution. 

Exercise 3: Show that (4.24) has a feasible solution iff y* = 0. 



4.4 LP Theory of a Firm in a Competitive Economy 
4.4.1 Activity analysis of the firm. 

We think of a firm as a system which transforms input into outputs. There are m kinds of inputs 
and k kinds of outputs. Inputs are usually classified into raw materials such as iron ore, crude oil, 
or raw cotton; intermediate products such as steel, chemicals, or textiles; capital goods such as 
machines of various kinds, or factory buildings, office equipment, or computers; finally various 
kinds of labor services. The firm's outputs themselves may be raw materials (if it is a mining 
company) or intermediate products (if it is a steel mill) or capital goods (if it manufactures lathes) 
or finished goods (if it makes shirts or bakes cookies) which go directly to the consumer. Labor is 
not usually considered an output since slavery is not practiced; however, it may be considered an 
output in a "closed," dynamic Malthusian framework where the increase in labor is a function of the 
output. (See the von Neumann model in (Nikaido [1968]), p. 141.) 

Within the firm, this transformation can be conducted in different ways, i.e., different combina- 
tions of inputs can be used to produce the same combination of outputs, since human labor can 
do the same job as some machines and machines can replace other kinds of machines, etc. This 
substitutability among inputs is a fundamental concept in economics. We formalize it by specifying 
which transformation possibilities are available to the firm. 

By an input vector we mean any m-dimensional vector r = (n, . . . , r m )' with r > 0, and by an 
output vector we mean any A; -dimensional vector y = (yi, . . . , y^)' with y > 0. We now make three 
basic assumptions about the firm. 

(i) The transformation of inputs into outputs is organized into a finite number, say n, of processes 
or activities. 

(ii) Each activity combines the k inputs in fixed proportions into the m outputs in fixed propor- 
tions. Furthermore, each activity can be conducted at any non-negative intensity or level. Pre- 
cisely, the jth activity is characterized completely by two vectors A 7 = (aij, d2j, • • • , a m j)' and 

= (hij, . . . , bkj)' so that if it is conducted at a level xj > 0, then it combines (transforms) the 
input vector (a\jXj, . . . , a m jXj)' = xjA^ into the output vector (byXj, . . . , b^xj)' = XjBK Let 

A be the m x n matrix [A 1 : . . . :A n ] and B be the k x n matrix B = [B 1 : . . . :B n ). 

3 It is more accurate to think of the services of capital goods rather than these goods themselves as inputs. It is these 
services which are consumed in the transformation into outputs. 




i=l 
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(iii) If the firm conducts all the activities simultaneously with the jth activity at level Xj > 0, 1 < j < 
n, then it transforms the input vector x\A l + . . . + x n A n into the output vector xiB 1 + . . . + x n B n . 

With these assumptions we know all the transformations technically possible as soon as we spec- 
ify the matrices A and B. Which of these possible transformations will actually take place depends 
upon their relative profitability and availability of inputs. We study this next. 

4.4.2 Short-term behavior. 

In the short-term, the firm cannot change the amount available to it of some of the inputs such as 
capital equipment, certain kinds of labor, and perhaps some raw materials. Let us suppose that these 
inputs are 1,2, ... ,£ and they are available in the amounts r*, . . . , r|, whereas the supply of the 
remaining inputs can be varied. We assume that the firm is operating in a competitive economy 
which means that the unit prices p = (pi, . . . ,Pk)' of the outputs, and q = (q\, . . . , q m )' of the 
inputs is fixed. Then the manager of the firm, if he is maximizing the firm's profits, faces the 
following decision problem: 



j=i+i 
subject to y = Bx, 

anxi + . . . + a in x n < r* , 1 < i < t , 
anx\ + . . . + a in x n < n , £ + 1 < i < m , 
xj>0, 1 < j < n; ri>0, £ + 1 < i < m . 



The decision variables are the activity levels x±, . . . ,x n , and the short-term input supplies r^ + \ , . . . , r. 
The coefficients of B and A are the fixed technical coefficients of the firm, the r* are the fixed short- 
term supplies, whereas the pi, qj are prices determined by the whole economy, which the firm ac- 
cepts as given. Under realistic conditions (4.33) has an optimal solution, say, x*, . . . , a;* , r| +1 , . . . ,r* 

4.4.3 Long-term equilibrium behavior. 

In the long run the supplies of the first £ inputs are also variable and the firm can change these 
supplies from r\, . . . , r| by buying or selling these inputs at the market price qi,...,q£. Whether 
the firm will actually change these inputs will depend upon whether it is profitable to do so, and in 
turn this depends upon the prices p, q. We say that the prices (p* , q* ) and a set of input supplies 
r* = {r\, . . . , rj^) are in (long-term) equilibrium if the firm has no profit incentive to change r* 
under the prices (p* , q*). 

Theorem 1: p* , q*,r* are in equilibrium if and only if q* is an optimal solution of (4.34): 



Proof: Let c = B'p*. By definition, p*,q*,r* are in equilibrium iff for all fixed A G R m , 
M(A) < M(0) where M(A) is the maximum value of the LP (4.35): 



rn 




Minimize (r*)'q 
subject to A'q > B'p* 
q > . 



(4.34) 



Maximize c'x — (g*)'A 
subject to Ax < r* + A , 
x > . 



(4.35) 
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For A = 0, (4.34) becomes the dual of (4.35) so that by the strong duality theorem, M (0) = {r*)'q*. 
Hence p*,q*, r* are in equilibrium iff 

dx - {q*)'A < M(0) = (r*)V , (4-36) 

whenever x is feasible for (4.35). By weak duality if x is feasible for (4.35) and q is feasible for 
(4.34), 

c'x - (q*)'A < q'{r* = A) - (q*)'A , (4.37) 
and, in particular, for q = q* , 

dx - (q*)'A < (q*)'{r* + A) - (q*)'A = (q*)'r* 

Remark 1: We have shown that (p*,q*,r* are in long-term equilibrium iff q* is an optimum 
solution to the dual (namely (4.34)) of (4.38): 

Maximize dx 

subject to Ax < r* (4.38) 
x > . 

This relation between p* ,q* ,r* has a very nice economic interpretation. Recall that c = B'p*, i.e., 
Cj = p*bij +py32j + • • • +P*kbkj. Now bij is the amount of the ith output produced by operating the 
jth activity at a unit level Xj = 1. Hence, Cj is the revenue per unit level operation of the jth activity 
so that dx is the revenue when the n activities are operated at levels x. On the other hand if the jth 
activity is operated at level Xj = 1, it uses an amount of the ith input. If the ith input is valued at 

m 

a*, then the input cost of operating at Xj = 1, is q%a,ij, so that the input cost of operating the n 

i=l 

activities at levels x is (A'q*)' = (q*)'Ax. Thus, if x* is the optimum activity levels for (4.38) then 
the output revenue is dx* and the input cost is (q*)'Ax*. But from (4.16), (q*)'(Ax* - r*) = so 
that 

c'x* = {q*)'r* , (4.39) 

i.e., at the optimum activity levels, in equilibrium, total revenues = total cost of input supplies. In 
fact, we can say even more. From (4.15) we see that if x = astj > then 

m 

Cj = Qi a ij ' 
i=l 

i.e., at the optimum, the revenue of an activity operated at a positive level = input cost of that activity. 
Also if 

m 
i=l 

then x* = 0, i.e., if the revenue of an activity is less than its input cost, then at the optimum it is 
operated at zero level. Finally, again from (4.15), if an equilibrium the optimum ith input supply r* 
is greater than the optimum demand for the ith input, 
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n 

r\ > ciijXj, 

then q* = 0, i.e., the equilibrium price of an input which is in excess supply must be zero, in other 
words it must be a free good. 

Remark 2: Returning to the short-term decision problem (4.33), suppose that 
(A*, . . . , A|, A| +1 , . . . , A£J is an optimum solution of the dual of (4.33). Suppose that the market 
prices of inputs 1, . . . ,£ are q\, . . . , qi. Let us denote by M(Ai, . . . , A^) the optimum value of 
(4.33) when the amounts of the inputs in fixed supply are rf + Ai, . . . , r| + A^. Then if 
(<9M/<9Aj)| a=o exists, we can see from (4.22) that it is always profitable to increase the ith input 
by buying some additional amount at price if A* > qi, and conversely it is profitable to sell some 
of the ith input at price qi if A* < qi. Thus A| can be interpreted as the firm's internal valuation of 
the ith input or the firm's imputed or shadow price of the ith input. This interpretation has wide 
applicability, which we mention briefly. Often engineering design problems can be formulated as 
LPs of the form (4.10) or (4.19), where some of the coefficients b{ are design parameters. The 
design procedure is to fix these parameters at some nominal value b* , and carry out the 
optimization problem. Suppose the resulting optimal dual variables are A*, then we see (assuming 
differentiability) that it is worth increasing b* if the unit cost of increasing this parameter is less 
than A* , and it is worth decreasing this parameter if the reduction in total cost per unit decrease is 
greater than A* . 



4.4.4 Long-term equilibrium of a competitive, capitalist economy. 

The profit-maximizing behavior of the firm presented above is one of the two fundamental building 
blocks in the equilibrium theory of a competitive, capitalist economy. Unfortunately we cannot 
present the details here. We shall limit ourselves to a rough sketch. We think of the economy as 
a feedback process involving firms and consumers. Let us suppose that there are a total of h com- 
modities in the economy including raw materials, intermediate and capital goods, labor, and finished 
products. By adding zero rows to the matrices (A, B) characterizing a firm we can suppose that all 
the h commodities are possible inputs and all the h commodities are possible outputs. Of course, 
for an individual firm most of the inputs and most of the outputs will be zero, the sole purpose for 
making this change is that we no longer need to distinguish between prices of inputs and prices of 
outputs. We observe the economy starting at time T. At this time there exists within the economy 
an inventory of the various commodities which we can represent by a vector lo = (u)±, . . . , u>h) > 0. 
(jj is that portion of the outputs produced prior to T which have not been consumed up to T. We are 
assuming that this is a capitalist economy, which means that the ownership of u is divided among 

the various consumers j = 1, . . . , J. More precisely, the jth consumer owns the vector of commodi- 

J 

ties u(j) > 0, and u(j) = to. We are including in u(j) the amount of his labor services which 

i=i 

consumer j is willing to sell. Now suppose that at time T the prevailing prices of the h commodities 
are A = (Ai, . . . , A/J' > 0. Next, suppose that the managers of the various firms assume that the 
prices A are not going to change for a long period of time. Then, from our previous analysis we 
know that the manager of the ith firm will plan to buy input supplies r(i) > 0, r(i) £ R h , such 
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that (A, r(i)) is in long term equilibrium, and he will plan to produce an optimum amount, say y(i). 
Here i = 1,2, ... , /, where I is the total number of firms. We know that r(i) and y(i) depend on 
A, so that we explicitly write r(i, A), y{i, A). We also recall that (see (4.38)) 

A'r(t, A) = X'y(i, A) , l<i<I. (4.40) 

Now the ith manager can buy r(i) from only two sources: outputs from other firms, and the con- 
sumers who collectively own uj. Similarly, the ith manager can sell his planned output y(i) either as 
input supplies to other firms or to the consumers. Thus, the net supply offered for sale to consumers 
is S(X), where 

J I i 

s(\) = <"0") + E f A ) - E r & A ) • ( 4 - 41 ) 

j=l i=l i=l 

We note two important facts. First of all, from (4.40), (4.41) we immediately conclude that 

J 

X'S(X) = J2Mj), (4-42) 

i=i 

that is the value of the supply offered to consumers is equal to the value of the commodities (and 
labor) which they own. The second point is that there is no reason to expect that 5(A) > 0. 

Now we come to the second building block of equilibrium theory. The value of the jth consumer's 
possessions is X'uj(j). The theory assumes that he will plan to buy a set of commodities d(j) = 
(di(j), . . . , dh(j)) > so as to maximize his satisfaction subject to the constraint X'd(j) = X'u(j). 
Here also d(j) will depend on A, so we write d(j, A). If we add up the buying plans of all the 
consumers we obtain the total demand 

J 

D(X) = Y,d(j,X) >0, (4.43) 

i=i 

which also satisfies 

J 

X'D(X) = A'w(j) . (4.44) 
i=i 

The most basic question of equilibrium theory is to determine conditions under which there exists a 
price vector Xe such that the economy is in equilibrium, i.e., S{Xe) = D{Xe), because if such an 
equilibrium price A# exists, then at that price the production plans of all the firms and the buying 
plan of all the consumers can be realized. Unfortunately we must stop at this point since we cannot 
proceed further without introducing some more convex analysis and the fixed point theorem. For 
a simple treatment the reader is referred to (Dorfman, Samuelson, and Solow [1958], Chapter 13). 
For a much more general mathematical treatment see (Nikaido [1968], Chapter V). 



4.5 Miscellaneous Comments 
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4.5.1 Some mathematical tricks. 

It is often the case in practical decision problems that the objective is not well-defined. There may 
be a number of plausible objective functions. In our LP framework this situation can be formulated 
as follows. The constraints are given as usual by Ax < b, x > 0. However, there are, say, k 
objective functions (c x )'x, . . . , (c k )'x. It is reasonable then to define a single objective function 
fo(x) by fo(x) = minimum {(c 1 )'^, (c 2 )'x, . . . , (c k )'x}, so that we have the decision problem, 

Maximize f Q (x) 

subject to Ax < b, x > . 

This is not a LP since /o is not linear. However, the following exercise shows how to transform 
(4.45) into an equivalent LP. 

Exercise 1: Show that (4.45) is equivalent to (4.46) below, in the sense that x* is optimal for (4.45) 
iff (x*,y*) = (x*,f (x*)) is optimal for (4.46). 

Maximize y 

subject to Ax < b, x < (4.46) 

y < (c^'x , 1 < i < k . 

Exercise 1 will also indicate how to do Exercise 2. 
Exercise 2: Obtain an equivalent LP for (4.47): 

n 

Maximize >c,(a; 7 ) 

f-f (4.47) 

subject to Ax < b, x < , 
where q : R — > R are concave, piecewise-linear functions of the kind shown in Figure 4.3. 

The above-given assumption of the concavity of the q is crucial. In the next exercise, the inter- 
pretation of "equivalent" is purposely left ambiguous. 

Exercise 3: Construct an example of the kind (4.47), where the Cj are piecewise linear (but not 
concave), and such that there is no equivalent LP. 

It turns out however, that even if the q are not concave, an elementary modification of the Simplex 
algorithm can be given to obtain a "local" optimal decision. See (Miller [1963]). 



4.5.2 Scope of linear programming. 

LP is today the single most important optimization technique. This is because many decision prob- 
lems can be adequately formulated as LPs, and, given the capabilities of modern computers, the 
Simplex method (together with its variants) is an extremely powerful technique for solving LPs in- 
volving thousands of variables. To obtain a feeling for the scope of LP we refer the reader to the 
book by one of the originators of LP (Dantzig [1963]). 
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Figure 4.3: A function of the form used in Exercise 2. 



Chapter 5 



OPTIMIZATION OVER SETS 
DEFINED BY INEQUALITY 
CONSTRAINTS: NONLINEAR 
PROGRAMMING 

In many decision-making situations the assumption of linearity of the constraint inequalities in LP 
is quite restrictive. The linearity of the objective function is not restrictive as shown in the first 
exercise below. In Section 1 we present the general nonlinear programming problem (NP) and 
prove the Kuhn-Tucker theorem. Section 2 deals with Duality theory for the case where appropriate 
convexity conditions are satisfied. Two applications are given. Section 3 is devoted to the important 
special case of quadratic programming. The last section is devoted to computational considerations. 

5.1 Qualitative Theory of Nonlinear Programming 

5.1.1 The problem and elementary results. 

The general NP is a decision problem of the form: 

Maximize f (x) 

subject to (x) < , i = 1, ... ,171, 

where x G R n , /j : R n — ► R, i = 0, 1, . . . , m, are differentiable functions. As in Chapter 4, 
x G R n is said to be a. feasible solution if it satisfies the constraints of (5.1), and SI C R n is the 
subset of all feasible solutions; x* G f2 is said to be an optimal decision or optimal solution if 
fo{x*) > fo( x ) for x G From the discussion in 4.1.2 it is clear that equality constraints and sign 
constraints on some of the components of x can all be transformed into the form (5.1). The next 
exercise shows that we could restrict ourselves to objective functions which are linear; however, we 
will not do this. 

Exercise 1: Show that (5.2), with variables y G R, x G R n , is equivalent to (5.1): 
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Maximize y 

subject to fi(x) < 0, 1 < i < m, and y — fo(x) < . 



(5.2) 



Returning to problem (5.1), we are interested in obtaining conditions which any optimal decision 
must satisfy. The argument parallels very closely that developed in Exercise 1 of 4.1 and Exercise 1 
of 4.2. The basic idea is to linearize the functions /j in a neighborhood of an optimal decision x*. 



Definition: Let a; be a feasible solution, and let I(x) C {1, . . . , m} be such that fi(x) = for 
i G I(x), fi(x) < for % G" I{x). (The set I{x) is called the set of active constraints at x.) 
Definition: (i) Let x G SI. A vector h G i? n is said to be an admissible direction for SI at x if there 
exists a sequence x k , fc = l,2,...,infi and a sequence of numbers e k , k = 1, . . . , with e fc > 
for all such that 



(ii) Let C(Sl, x) = is an admissible direction for SI at x}. C(Sl, x) is called the tangent cone 
of SI at x. Let K(Sl, x) = {x + h\h G C(fi, a;)}. (See Figures 5.1 and 5.2 and compare them with 
Figures 4.1 and 4.2.) 

If we take x k = x and e k = 1 for all /c, we see that £ C(f2, x) so that the tangent cone is always 
nonempty. Two more properties are stated below. 

Exercise 2: (i) Show that C(0, x) is a cone, i.e., if /i G C(fi, x) and > 0, then 9h G C(J7, x). 
(ii) Show that C(0, x) is a closed subset of i? n . (Hint for (ii): For m = 1, 2, . . . , let h m and 

{x mk , e mk > 0}^i 1 be such that x mk — > x and (l/e mfc )(x mfc — x) — >■ h m as fc — > oo. Suppose 
that h m — >• /i as m — > oo. Show that there exist subsequences {x mkm , e mkm }^ =1 such that 
x mfcm — ► x and (l/£ Tnfc m)( a ; mfc " J — x) — >• h as m — > oo.) 

In the definition of C(0, x) we made no use of the particular functional description of Si. The 
following elementary result is more interesting in this light and should be compared with (2.18) in 
Chapter 2 and Exercise 1 of 4.1. 



Lemma 1: Suppose x* G Si is an optimum decision for (5.1). 



lim 



k 

X = X 



limM xk - x ) = h ■ 



Then 



hx{x*)h < for all h G C{Sl, x*) . 



(5.3) 



Proof: Let x k G Si, e k > 0, A; = 1,2,3,... 



be such that 
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direction of 




Figure 5.1: O = PQR 



lim xk=x * > lim M x "- X *) = h ■ 

k— >oo fc— too 

Note that in particular (5.4) implies 

lim & \x k ~x*\ = \h\ . 

fc-^oo 

Since /o is differentiable, by Taylor's theorem we have 

k{x k ) = fo(x* + (x fc - x*)) = f (x*) + f 0x (x*)(x k - x*) + o{\x k -x*\). 
Since x k E $7, and x* is optimal, we have fo(x k ) < /o(x*), so that 

Taking limits as k — > oo, using (5.4) and (5.5), we can see that 



> 



lim 



fox(x*) 



(x k —x*) 



+ 



lim 



o(\x k — x*\) 
\x k —x*\ 



lim 



las* 



f0x(x*)h. 
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The basic problem that remains is to characterize the set C(Q, x*) in terms of the derivatives of the 
functions /j. Then we can apply Farkas' Lemma just as in Exercise 1 of 4.2. 

Lemma 2: Let x* G IX Then 

C(U, x*) C {h\f ix (x*)h < for all i G I{x*)} . (5.7) 

Proof: Let h G i? n and x fc G $7, e fc > 0, fc = 1, 2, . . . , satisfy (5.4). Since is differentiable, by 
Taylor's theorem we have 

fi(x k ) = fi(x*) + f ix {x*){x k - x*) + o{\x k -x*\). 

Since x fe G 17, /i(x fc ) < 0, and if i G /i(x*) = 0, so that fi(x h ) < fi(x*). Following the 

proof of Lemma 1 we can conclude that > fi x {x*)h. 

Lemma 2 gives us a partial characterization of C(fi, x*). Unfortunately, in general the inclusion 
sign in (5.7) cannot be reversed. The main reason for this is that the set {fi x {x*)\i G I{x*)} is not 
in general linearly independent. 

Exercise 3: Let x G R 2 , fi(x\,X2) = (x\ — l) 3 + X2, and f2(xi,X2) = —X2- Let 
(x\,x* 2 ) = (1,0). Thenl(x*) = {1,2}. Show that 
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C(n,x*)^{h\f ix (x*)h<0 , z = l,2,}. 
(Note that {fi x (x*), f2x{x*)} is not a linearly independent set; see Lemma 4 below.) 

5.1.2 Kuhn-Tucker Theorem. 

Definition: Let x* G fl We say that the constraint qualification (CQ) is satisfied at x* if 

C(Q,x) = {h\f ix (x*)h < OforalH G I(x*)}, 

and we say that CQ is satisfied if CQ is satisfied at all x G IX (Note that by Lemma 2 C(0, rr) is 
always a subset of the right-hand side.) 

Compare the next result with Exercise 2 of 4.2. 

Theorem 1: (Kuhn and Tucker [1951]) Let x* be an optimum solution of (5.1), and suppose that 
CQ is satisfied at x* . Then there exist A* > 0, for i G I(x*), such that 

fox(x*) = Kfix{x*) (58) 

i&I(x*) 

Proof: By Lemma 1 and the definition of CQ it follows that fo £ {x*)h < whenever fi x (x*)h < 
for all i G I(x*). By the Farkas' Lemma of 4.2.1 it follows that there exist A* > for i G I{x*) 
such that (5.8) holds. 

In the original formulation of the decision problem we often have equality constraints of the form 
Tj(x) = 0, which get replaced by rj(x) < 0, —Vj(x) < to give the form (5.1). It is convenient in 
application to separate the equality constraints from the rest. Theorem 1 can then be expressed as 
Theorem 2. 



Theorem 2: Consider the problem (5.9). 



Maximize fo(x) 

subject to fi(x) < , i = 1, . . . , m, (5.9) 
r j( x ) = , j = 1, • • • ,k . 

Let x* be an optimum decision and suppose that CQ is satisfied at x*. Then there exist A* > 0, i = 
1, . . . , m, and n*, j = 1, . . . , k such that 

m k 

fox(x*) = Khx(x*) + 1<) <>(■<■■) , (5.10) 

i=l j=l 

and 

A* = whenever fi{x*) < . (5.11) 

Exercise 4: Prove Theorem 2. 
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An alternative form of Theorem 1 will prove useful for computational purposes (see Section 4). 
Theorem 3: Consider (5.9), and suppose that CQ is satisfied at an optimal solution x*. Define 

V> : R n ' -> R by 

ijj{h) = max {-f 0x {x*)h, fi(x*) + fi x (x*)h, f m (x*) + f mx (x*)h} , 
and consider the decision problem 

Minimize ip(h) 

subject to -ip(h) - f 0x (x*)h < 0, 

-^(fc) + ft(x*) + / iiE (x*)fe < , 1 < i < m KD - } 

— 1 < hi < 1 , i = 1, . . . , n . 

Then = is an optimal solution of (5.12). 

Exercise 5: Prove Theorem 3. (Note that by Exercise 1 of 4.5, (5.12) can be transformed into a 
LP.) 

Remark: For problem (5.9) define the Lagrangian function L: 

m k 

(xt,.. . ,x n ; Ai,. . . , A m ;/xi,. . . i-> / (x) - ^ Ai/i(x) - ^^^(x). 

i=i i=i 

Then Theorem 2 is equivalent to the following statement: if CQ is satisfied and x* is optimal, then 
there exist A* > and /j,* such that L x (x*, X*,fi*) = and L(x*,X*,fi*) < L(x* , A, fi) for all 
A > 0,/i. 

There is a very important special case when the necessary conditions of Theorem 1 are also 
sufficient. But first we need some elementary properties of convex functions which are stated as an 
exercise. Some additional properties which we will use later are also collected here. 
Recall the definition of convex and concave functions in 4.2.3. 

Exercise 6: Let X C R n be convex. Let h : X — > R be a differentiable function. Then 

(i) h is convex iff h(y) > h(x) + h x (x)(y — x) for all x, y, in X, 

(ii) h is concave iff h(y) < h(x) + h x (x)(y — x) for all x, y in X, 

(iii) h is concave and convex iff h is affine, i.e. h(x) = a + b'x for some 
fixed ae R,be R n . 

Suppose that h is twice differentiable. Then 

(iv) h is convex iff h xx (x) is positive semidefinite for all x in X, 

(v) h is concave iff h xx (x) is negative semidefinite for all x in X, 

(vi) h is convex and concave iff h xx (x) = 0. 

Theorem 4: (Sufficient condition) In (5.1) suppose that fo is concave and is convex for 
i = 1, . . . , m. Then 

(i) O is a convex subset of R n , and 

(ii) if there exist x* G 0, A* > 0, i G I{x*), satisfying (5.8), then x* is an optimal solution of 
(5.1). 

Proof: 

(i) Let y, z be in Q so that fi(y) < 0, fi(z) < for i = 1, . . . , m. Let < 6 < 1. Since /j is 
convex we have 
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fi(9y + (1 - 9)z) < 9fi(y) + (1 - 9)fi{z) < , 1 < i < m, 

so that (9y + (1 — G $7, hence O is convex. 

(ii) Let x G $7 be arbitrary. Since /o is concave, by Exercise 6 we have 

fo(x) < f (x*) + f 0x (x*)(x - X*) , 

so that by (5.8) 

/o(a:) < fo(x*) + Khx{x*){x - x*) . (5 13) 

iei(x*) 

Next, fi is convex so that again by Exercise 6, 

fi(x) > ft(x*) + f ix (x*)(x - x*) ; 

but fi(x) < 0, and fi(x*) = for i G J(s*), so that 

- x*) < for i G J(x*) . (5.14) 

Combining (5.14) with the fact that A* > 0, we conclude from (5.13) that fo(x) < fo(x*), so that 
x* is optimal. 
Exercise 7: Under the hypothesis of Theorem 4, show that the subset 17* of O, consisting of all the 
optimal solutions of (5.1), is a convex set. 

Exercise 8: A function h : X — > R defined on a convex set X C R n is said to be strictly convex if 
h(9y + (1 - 9)z) < 9h(y) + (1 - 9)h(z) whenever < 9 < 1 and y, z are in X with y^z.hte 
said to be strictly concave if — h is strictly convex. Under the hypothesis of Theorem 4, show that 
an optimal solution to (5.1) is unique (if it exists) if either /o is strictly concave or if the 
fi, 1 < i < Tn, are strictly convex. (Hint: Show that in (5.13) we have strict inequality if x ^ x* .) 



5.1.3 Sufficient conditions for CQ. 

As stated, it is usually impractical to verify if CQ is satisfied for a particular problem. In this 
subsection we give two conditions which guarantee CQ. These conditions can often be verified in 
practice. Recall that a function g : R n — ► R is said to be qffine if g(x) = a + b'x for some fixed 
a G R and b G R n . 

We adopt the formulation (5.1) so that 

n = {x G R n \fi{x) < , 1 < i < m} . 

Lemma 3: Suppose and suppose there exists h* G R n such that for each i G I(x*), either 

fi x (x*)h* < 0, or fi x (x*)h* = and /, is affine. Then CQ is satisfied at x*. 
Proof: Let h G R n be such that f ix (x*)h < for i G /(x*). Let 5 > 0. We will first show that 
(h + 5h*) G C(Q, x*). To this end let e k > 0, A; = 1, 2, . . . , be a sequence converging to and set 
x k = x* + e fc (/i + J/i*). Clearly x k converges to x*, and (l/e k )(x k — x*) converges to (h + 5h*). 
Also for i G I(x*), if fi x (x*)h < 0, then 

/i(x fc ) = fi(x*) + e fc /«(s*)(/i + + o(e fc |/i + ^*|) 

< 5e k f ix (x*)h* + o(e k \h + 5h*\) 

< for sufficiently large k , 

whereas for i G /(x*), if fi is affine, then 
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fi(x k ) = fi(x*) + e k f lx (x*)(h + Sh*) < for all k . 

Finally, for i G" I(x*) we have f%(x*) < 0, so that fi(x k ) < for sufficiently large k. Thus we 
have also shown that x k G $7 for sufficiently large k, and so by definition (h + Sh*) G C(Q, x*). 
Since S > can be arbitrarily small, and since C(Cl, x*) is a closed set by Exercise 2, it follows that 

heC(n,x*). 

Exercise 9: Suppose i* ell and suppose there exists x G i? n such that for each i G I{x*), either 
fi(x*) < and /j is convex, or /j(x) < and f\ is affine. Then CQ is satisfied at x* . (Hint: Show 
that h* = x — x* satisfies the hypothesis of Lemma 3.) 

Lemma 4: Suppose i* eSJ and suppose there exists h* G R n such that fi x (x*)h* < for 
i G I(x*), and {/^(x*)^ G I{x*), fi x (x*)h* = 0} is a linearly independent set. Then CQ is 
satisfied at x* . 

Proof: Let /i G R n be such that f ix {x*)h < for all i G I(x*). Let 5 > 0. We will show that 
{h + (5/i*) G C(U,x*). Let J 5 = {i\i G /(x*), fi x {x*){h + <J/i*) = 0}, consist of p elements. 
Clearly J$ C J = G /(a;*), fix(x*)h* = 0}, so that {/^(x*, n*)|i G J^} is linearly 
independent. By the Implicit Function Theorem, there exist p > 0, an open set V C R n containing 
x* = (w*,u*), and a differentiable function g : U — > where ?7 = {u G i? n ~ p ||u — u*| < p}, 
such that 



fi(w,u) = 0, i G J<5, and (it;, u) G V 



iff 



u G U, and io = g(u) . 

Next we partition h, h* as h = 77), /i* = (£*, 77*) coiTesponding to the partition of x = (to, u). 
Let e fc > 0, = 1, 2 . . . , be any sequence converging to 0, and set u k = u* + e k {rj + 5rf), w k = 
g(u k ), and finally x fe = (s k ,u k ). 

We note that u fc converges to u* , so 7y fc = g{u k ) converges to w* = g(u*). Thus, x k converges 
to x*. Now (l/e k )(x k — x*) = (l/e k )(w k — w*,u k — u*) = (l/e k )(g(u k ) — g(u*), e k {rj + ^77*)). 
Since g is differentiable, it follows that (l/e fc )(x fc — x*) converges to (g u {u*)(rj + 8rj*), r] + Sr}*). 
But for i G J$ we have 

= f ix {x*){h + 5h*) = f iw {x*){i + 6C) + fiu(x*)(v + S V *) . (5.15) 

Also, for i G J$, = fi{g{u),u) for u € U so that = fi W (x*)g u (u*) + fiu(x*), and hence 

= /™(x*) 5u (n*)(r/ + 57 ? *) + / iu (x*)(r/ + ( 5r/*) . (5.16) 

If we compare (5.15) and (5.16) and recall that {/««,(x*)|i G Js} is a basis in we can conclude 
that (£ + = g u {u*){rj + 5rf) so that (l/e fc )(x fc - x*) converges to (/i + hSh*). 

It remains to show that x fc G fl for sufficiently large fc. First of all, for i G J5, fi(x k ) = 
fi(g(u k ),u k ) = 0, whereas fori $ Js, i e I{x*), 

fi(x k ) = fi(x*) + f ix (x*)(x k - x*) + o(\x k - x*\) 

fi(x*) + e k f ix (x*)(h + Sh*) + o(e k ) + o(\x k - x*\), 
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and since f%(x*) = whereas fi x (x*)(h + 5h*) < 0, we can conclude that fi(x k ) < for suffi- 
ciently large k. Thus, x k G O for sufficiently large k. Hence, (h + 5h*) G C(Q, x*). 

To finish the proof we note that 5 > can be made arbitrarily small, and C(Q, x*) is closed by 
Exercise 2, so that h G C(Q, x*). 

The next lemma applies to the formulation (5.9). Its proof is left as an exercise since it is very 
similar to the proof of Lemma 4. 

Lemma 5: Suppose x* is feasible for (5.9) and suppose there exists h* G R n such that the set 
{fix(x*)\i G I(x*)J ix (x*)h* = 0}\J{r jx (x*)\j = 1, . . . , k} is linearly independent, mdf ix (x*)h* < 
for i G I(x*), r jx (x*)h* = for 1 < j < k. Then CQ is satisfied at x*. 

Exercise 10: Prove Lemma 5 



5.2 Duality Theory 

Duality theory is perhaps the most beautiful part of nonlinear programming. It has resulted in many 
applications within nonlinear programming, in terms of suggesting important computational algo- 
rithms, and it has provided many unifying conceptual insights into economics and management 
science. We can only present some of the basic results here, and even so some of the proofs are 
relegated to the Appendix at the end of this Chapter since they depend on advanced material. How- 
ever, we will give some geometric insight. In 2.3 we give some application of duality theory and in 
2.2 we refer to some of the important generalizations. The results in 2. 1 should be compared with 
Theorems 1 and 4 of 4.2.1 and the results in 4.2.3. 

It may be useful to note in the following discussion that most of the results do not require differ- 
entiability of the various functions. 

5.2.1 Basic results. 

Consider problem (5.17) which we call the primal problem: 

Maximize fo(x) 

subject to fi(x) < bi , 1 < i < m (5-17) 

iel , 

where x G R n , fi : R n — > R, 1 < i < m, are given convex functions, /o : R n — > R is a 
given concave function, X is a given convex subset of R n and b = (b\, . . . , b m )' is a given vector. 
For convenience, let / = (fx, . . . , / m )' : R n — > R m . We wish to examine the behavior of the 
maximum value of (5.17) as b varies. So we define 

n(b) = {x\x G X, f(x) < 6}, B = {b\U(b) / <f>}, 

and 

M: R{J{+oo} by M(b) = suy{f (x)\x G XJ(x) < b} 

= sup{/ (x)|x G n(b)} , 

so that in particular if x* is an optimal solution of (5.17) then M(b) = fo(x). We need to consider 
the following problem also. Let A G R m , A > 0, be fixed. 



Maximize f (x) - \'{f(x) - b) 
subject to x G X , 



(5.18) 
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and define 

m(A) = sub{/ (x) - A'(/(V) - b)\x G X} . 
Problem (5.19) is called the dual problem: 

Minimize m(A) 
subject to A > . 

Let m* =inf {m(A)|A > 0}. 

Remark 1: The set X in (5.17) is usually equal to R n and then, of course, there is no reason to 

separate it out. However, it is sometimes possible to include some of the constraints in X in such 

a way that the calculation of m(A) by (5.18) and the solution of the dual problem (5.19) become 

simple. For example see the problems discussed in Sections 2.3.1 and 2.3.2 below. 

Remark 2: It is sometimes useful to know that Lemmas 1 and 2 below hold without any convexity 

conditions on /q, /, X. Lemma 1 shows that the cost function of the dual problem is convex which 

is useful information since there are computation techniques which apply to convex cost functions 

but not to arbitrary nonlinear cost functions. Lemma 2 shows that the optimum value of the dual 

problem is always an upper bound for the optimum value of the primal. 

Lemma 1: m : — > R\J{+oo} is a convex function. (Here R^_ = {A G R n \\ > 0}.) 

Exercise 1: Prove Lemma 1. 

Lemma 2: (Weak duality) If x is feasible for (5.17), i.e., x G S7(6) , and if A > 0, then 

fo(x) < M(b) < m* < m(A) . (5.20) 
Proof: Since f(x) - b < 0, and A > 0, we have \'{f{x) - b) < 0. So, 

fo{x) < f Q (x) - X'(f(x) — b) , for x G O(S), A > . 

Hence 

f (x) < sup {f (x)\x G fi(S)} = M(b) 

< sup {f (x) - A'(/(x) — b)\x£ O(S)} and since 0(6) C X, 

< sup {f (x) - X'(f(x) - b)\x G X} = m(A) . 

Thus, we have 

f (x) < M(b) < m(A) for x G A > , 

and since M{b) is independent of A, if we take the infimum with respect to A > in the right-hand 
inequality we get (5.20). 
The basic problem of Duality Theory is to determine conditions under which M(b) = m* in 
(5.20). We first give a simple sufficiency condition. 

Definition: A pair (x, A) with x G X, and A < is said to satisfy the optimality conditions if 
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x is optimal solution of (5.18) with A = A, (5-21) 

x is feasible for (5.17), i.e., fi(x) < b\ for i = 1, . . . , m , (5-22) 

Aj = when /i(x) < 6j, equivalently , A'(/(x) — 6) = 0. (5.23) 



A > is said to be an optimal price vector if there is x G X such that (x, A) satisfy the optimality 
condition. Note that in this case x, G $7(6) by virtue of (5.22). 

The next result is equivalent to Theorem 4(ii) of Section 1 if X = R n , and fi, < i < m, are 
differentiable. 

Theorem 1: (Sufficiency) If (x, A) satisfy the optimality conditions, then x is an optimal solution to 
the primal, A is an optimal solution to the dual, and M{b) = m* . 
Proof: Let x G 0(6), so that A'(/(x) - 6) < 0. Then 

fo(x) < f (x) - X'(f(x) - 6) 

< sup{/ (x)-A'(/(x)-S)|x€X} 
= fo(x) - X'(f(x) - 6) by (5.21) 
= f (x) by (5.23) 

so that x, is optimal for the primal, and hence by definition fo(x) = M(b). Also 

m(A) = f (x) - \'(f(x) - 6) 
fo(x) = M(b) , 

so that from Weak Duality A is optimal for the dual. 

We now proceed to a much more detailed investigation. 
Lemma 3: B is a convex subset of R m , and M : B — > R U{+°°} i s a concave function. 
Proof: Let 6, 6 belong to B, let x G 0(6), x G 0(6), let < 6 < 1. Then (0x + (1 - 0)x) 
since X is convex, and 

fi(6x + (1 - 0)x) < 0/i(x) + (1 - 9)fi(x) 

since /i is convex, so that 

fi(9x + (1 - 9)x) <9b+{l- 9)1 , (5.24) 

hence 

{9x + (1 - 0)x) G n{9b + (1 - 0)6) 

and therefore, B is convex. 
Also, since fo is concave, 

/o(0x + (1 - 9)x) > 9f (x) + (1 - 0)/ o (x) . 




G X 
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Since (5.24) holds for all x G 0(6) and x G 0(6) it follows that 

M(6b + (1 - 0)6) > sup {f (9x + (1 - 0)x)|x G 0(6), x G 0(6)} 

> sup{/ o (x)|xGO(6)} + (l-0) sup {fo(x)\x G 0(6)} 
= 0M(6) + (1 - 0)M(6). 

Definition: Let X C R n and let 5 : X — > R \j{oo, —00}. A vector A G R n is said to be a 
supergradient (subgradient) of g at x G X if 

S'(^) < g(x) + A'(x — £) for x G X. 
(S'(^) > #(£) + A'(x - x) for x G X.) 

(See Figure 5-3.) 



M(6) 



M(6) 



b£B b 

M is not stable at 6 



M(6) 



M(6) 




M is stable at 6 

__M(6) + A'(6 - 6) 




6 6 
A is a supergradient at 6 
Figure 5.3: Illustration of supergradient of stability. 



Definition: The function M : B — > i? U{°°} i s sa id to be stable at 6 G i? if there exists a real 
number A' such that 

M(6) < M(6) + K|6 - 6| for 6 G B . 

(In words, M is stable at 6 if M does not increase infinitely steeply in a neighborhood of 6. See 
Figure 5.3.) 

A more geometric way of thinking about subgradients is the following. Define the subset A C 

R l+m by 



5.2. DUALITY THEORY 



61 



A = {(r,b)\b G B, and r < M(6)} . 

Thus A is the set lying "below" the graph of M. We call A the hypograph 1 of M. Since M is 
concave it follows immediately that A is convex (in fact these are equivalent statements). 

Definition: A vector (Ao, Ai, . . . , A m ) is said to be the normal to a hyperplane supporting A at a 
point(f, b) if 



A r + x i*>i > x o r + Yl Xibi for a11 ( r ' 6 ) G A ■ ( 5 - 25 ) 

j=i i=i 

(In words, ^4 lies below the hyperplane 7r = {(r, 6)|Aor+^ Aj&j = Aof +^ Aj&j}.) The supporting 
hyperplane is said to be non-vertical if Ao / 0. See Figure 5.4. 

Exercise 2: Show that if b G B, b > b, and f < M(S), then b £ B, M(b), and (f, 6) G A. 
Exercise 3: Assume that b £ B, and M(6) < oo. Show that (i) if A = (Ai, . . . , A m )' is a 
supergradient of M at b then A > 0, and (1, — Ai, . . . , — X m )' defines a non-vertical hyperplane 
supporting A at (M(6), b), (ii) if (Ao, — Ai, . . . , —Xm)' defines a hyperplane supporting A at 
(M(b), b) then Ao > 0, Aj > for 1 < i < m; futhermore, if the hyperplane is non- vertical then 
((Ai/Ao, • • • , (A m /Ao))' is a supergradient of M at b. 

We will prove only one part of the next crucial result. The reader who is familiar with the 
Separation Theorem of convex sets should be able to construct a proof for the second part based 
on Figure 5.4, or see the Appendix at the end of this Chapter. 

Lemma 4: (Gale [1967]) M is stable at b iff M has a supergradient at b. Proof: (Sufficiency only) 
Let A be a supergradient at b, then 

M{b) < M{b) + A'O - b) 

< M{b) + \\\\b-b\ . 

The next two results give important alternative interpretations of supergradients. 
Lemma 5: Suppose that x is optimal for (5.17). Then A is a supergradient of M at b iff A is an 
optimal price vector, and then (x, A) satisfy the optimality conditions. 

Proof: By hypothesis, f(x) = M(b), x G X, and f(x) < b. Let A be a supergradient of M at b. 
By Exercise 2, (M(b), f(x)) G A and by Exercise 3, A > and 

M{b) - X'b > M{b) - \'f{x) , 

so that A'(/(x) - b) > 0. But then A'(S - f(x)) = 0, giving (5.23). Next let x G X. Then 
ifo( x ), f( x )) G A, hence again by Exercise 3 

M (b) - X'b > f (x) - X'f{x) . 

Since fo(x) = M{b), and X'(f(x) — b) = 0, we can rewrite the inequality above as 

'From the Greek "hypo" meaning below or under. This neologism contrasts with the epigraph of a function which is 
the set lying above the graph of the function. 
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M{b) 



M{b) 




No non-vertical hyperplane supporting A at (M(6), b) 
M{b) 




(Ao, 



b b 
tt is a non- vertical hyperplane supporting A at (M(b),b) 

Figure 5.4: Hypograph and supporting hyperplane. 



/o(x) + A'(/(x) - S) > f (x) - X'(f(x) - b) , 

so that (5.21) holds. It follows that (x, A) satisfy the optimality conditions. 

Conversely, suppose x € X, A > satisfy (5.21), (5.22), and (5.23). Let x £ £l(b), i.e., 
xeX, f(x) < b. Then A'(/(x) - b) < so that 



fo(x) < f (x) X'(f(x) - b) 

= fo(x) - X'(f(x) -b) + X'(b - b) 
< f (x) - X'(f(x) -b) + X'(b - b) 
= f (x) + X'(b - b) 
= M(b) + X'(b-b) . 



by (5.21) 
by (5.23) 



Hence 



M{b) = sup{/ (x)|x G n(b)} < M(b) + X'(b - b) , 



so that A' is a supergradient of M at b. 
Lemma 6: Suppose that b G B, and M{b) < oo. Then A is a supergradient of M at b iff A is an 
optimal solution of the dual (5.19) and m{X) = M(b). 
Proof: Let A be a supergradient of M at b. Let x £ X. By Exercises 2 and 3 
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M(S) - X'b > f (x) - X'f(x) 

or 

M(S) > /o(x) - X'(f(x) - b) , 

so that 

M(S) > sup{/ (x) - £'(/(*) - 6)|x G X} = m(A) . 

By weak duality (Lemma 2) it follows that M(b) = m(A) and A is optimal for (5.19). 
Conversely suppose A > 0, and m(A) = M(b). Then for any i£l 

M(S) > f (x) - X'(f(x) - b) , 

and if moreover f(x) < b, then A'(/(x) — b) < 0, so that 

M(S) > / (x) - X'(f(x) - b) + X'(f(x) - b) 
= f {x) - X'b + X'b for x G 0(6) . 

Hence, 

M(6) = sup{/ (x)|x G n(b)} < M{b) + X'(b - b) , 

so that A is a supergradient. 

We can now summarize our results as follows. 
Theorem 2: (Duality) Suppose I )6 B, M(b) < oo, and M is stable at b. Then 

(i) there exists an optimal solution A for the dual, and m(A) = M(b), 

(ii) A is optimal for the dual iff A is a supergradient of M at b, 

(iii) if A is any optimal solution for the dual, then x is optimal for the primal iff (£, A) satisfy the 
optimality conditions of (5.21), (5.22), and (5.23). 

Proof: (i) follows from Lemmas 4,6. (ii) is implied by Lemma 6. The "if" part of (iii) follows from 
Theorem 1, whereas the "only if" part of (iii) follows from Lemma 5. 
Corollary 1: Under the hypothesis of Theorem 2, if A is an optimal solution to the dual then 

(dM+/dbi)(b) <Xi< (dM-/dbi)(b). 

Exercise 4: Prove Corollary 1. (Hint: See Theorem 5 of 4.2.3.) 
5.2.2 Interpretation and extensions. 

It is easy to see using convexity properties that, if X = R n and fi , < i < m, are differentiable, 
then the optimality conditions (5.21), (5.22), and (5.23) are equivalent to the Kuhn-Tucker condition 
(5.8). Thus the condition of stability of M at b plays a similar role to the constraint qualification. 
However, by Lemmas 4, 6 stability is equivalent to the existence of optimal dual variables, whereas 
CQ is only a sufficient condition. In other words if CQ holds at x then M is stable at b. In particular, 
if X = R n and the fi are differentiable, the various conditions of Section 1.3 imply stability. Here 
we give one sufficient condition which implies stability for the general case. 

Lemma 7: If b is in the interior of B, in particular if there exists x G X such that fi{x) < b\ for 
1 < i < m, then M is stable at b. 
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The proof rests on the Separation Theorem for convex sets, and only depends on the fact that M 
is concave, M(b) < oo without loss of generality, and b is the interior of B. For details see the 
Appendix. 

Much of duality theory can be given an economic interpretation similar to that in Section 4.4. 
Thus, we can think of x as the vector of n activity levels, fo(x) the corresponding revenue, X as 
constraints due to physical or long-term limitations, b as the vector of current resource supplies, 
and finally f(x) the amount of these resources used up at activity levels x. The various convexity 
conditions are generalizations of the economic hypothesis of non-increasing returns-to-scale. The 
primal problem (5.17) is the short-term decision problem faced by the firm. Next, if the current 
resources can be bought or sold at prices A = (Ai, . . . , A m )', the firm faces the decision problem 
(5.18). If for a price system A, an optimal solution of (5.17) also is an optimal solution for (5.18), 
then we can interpret A as a system of equilibrium prices just as in 4.2. Assuming the realistic 
condition b G B, M(b) < oo we can see from Theorem 2 and its Corollary 1 that there exists 
an equilibrium price system iff (dM + /dbi)(b) < oo, 1 < i < m; if we interpret (dM + /dbi)(b) 
as the marginal revenue of the ith resource, we can say that equilibrium prices exist iff marginal 
productivities of every (variable) resource is finite. These ideas are developed in (Gale [1967]). 



M{b) 




Figure 5.5: If M is not concave there may be no supporting hyperplane at (M(b),b). 

Referring to Figure 5.3 or Figure 5.4, and comparing with Figure 5.5 it is evident that if M is not 
concave or, equivalently, if its hypograph A is not convex, there may be no hyperplane supporting A 
at (M(b), b). This is the reason why duality theory requires the often restrictive convexity hypoth- 
esis on X and /j. It is possible to obtain the duality theorem under conditions slightly weaker than 
convexity but since these conditions are not easily verifiable we do not pursue this direction any fur- 
ther (see Luenberger [1968]). A much more promising development has recently taken place. The 
basic idea involved is to consider supporting A at (M(b), b) by (non- vertical) surfaces tt more gen- 
eral than hyperplanes; see Figure 5.6. Instead of (5.18) we would then have more general problem 
of the form (5.26): 



Maximize f (x) - F(f(x) - b) 
subject to x e X , 



(5.26) 
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where F : R m — > R is chosen so that 7r (in Figure 5.6) is the graph of the function b h-> M(b) — 
F(b — b). Usually F is chosen from a class of functions cj> parameterized by fj, = (/ii, . . . , > 0. 
Then for each fixed /i>0we have (5.27) instead of (5.26): 

Maximize f (x) - <f>{(i; f{x) - b) _ 
subject to x G X . 




Figure 5.6: The surface tt supports A at (M(b), b). 



If we let 

=sup{/ (x) - /(s) - S)|x G X} . 

then the dual problem is 

Minimize 
subject to fx > , 

in analogy with (5.19). 

The economic interpretation of (5.27) would be that if the prevailing (non-uniform) price system 
is </>(//; •) then the resources f(x) — b can be bought (or sold) for the amount </>(//; f(x) — b). For 
such an interpretation to make sense we should have b) > for b > 0, and 0(/x; b) > 0(/x; 6) 
whenever 6 > b. A relatively unnoticed, but quite interesting development along these lines is 
presented in (Frank [1969]). Also see (Arrow and Hurwicz [I960]). 

For non-economic applications, of course, no such limitation on <fi is necessary. The following 
references are pertinent: (Gould [1969]), (Greenberg and Pierskalla [1970]), (Banerjee [1971]). For 
more details concerning the topics of 2.1 see (Geoffrion [1970a]) and for a mathematically more 
elegant treatment see (Rockafellar [1970]). 

5.2.3 Applications. 
Decentralized resource allocation. 

Parts (i) and (iii) of Theorem 2 make duality theory attractive for computation purposes. In particular 
from Theorem 2 (iii), if we have an optimal dual solution A then the optimal primal solutions are 
those optimal solutions of (5.18) for A = A which also satisfy the feasibility condition (5.22) and 
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the "complementary slackness" condition (5.23). This is useful because generally speaking (5.18) 
is easier to solve than (5.17) since (5.18) has fewer constraints. 

Consider a decision problem in a large system (e.g., a multi-divisional firm). The system is 
made up of k sub-systems (divisions), and the decision variable of the ith sub-system is a vector 
x % G R ni , 1 < i < k. The sub-system has individual constraints of the form x % G X 1 where x % is 
a convex set. Furthermore, the sub-systems share some resources in common and this limitation is 
expressed as fix 1 ) + . . . + f k (x k ) < b where /* : R ni — > R m are convex functions and b G R m 
is the vector of available common resources. Suppose that the objective function of the large system 
is additive, i.e. it is the form /^(x 1 ) + . . . + f$(x k ) where /q : R ni — > R are concave functions. 
Thus we have the decision problem (5.28): 

k 

Maximize fix 1 ) 
i=i 

subject to x i G X i , 1 < i < k, (5.28) 

k 

£/V)<& • 

i=l 

For A G R m , A > 0, the problem corresponding to (5.19) is 

k 

Maximize f^x*) - X' fix 1 ) - A'(^ fix 1 ) - b) 

i=i 

subject to x % G X 1 , 1 < z < A; , 



which decomposes into k separate problems: 



Maximize fix -1 ) - X' fix 1 ) 
subject to x % G X\ , 1 < i < k 



(5.29) 



If we let m*(A) = supj/^^) - A7 i (x i )|x i G X% and m(A) = J^m^A) + X'b, then the dual 
problem is 



i=i 



Minimize m(A) 
subject to A > 



(5.30) 



Note that (5.29) may be much easier to solve than (5.28) because, first of all, (5.29) involves fewer 
constraints, but perhaps more importantly the decision problems in (5.29) are decentralized whereas 
in (5.28) all the decision variables x 1 , . . . , x k are coupled together; in fact, if k is very large it may 
be practically impossible to solve (5.28) whereas (5.29) may be trivial if the dimensions of x l are 
small. 

Assuming that (5.28) has an optimal solution and the stability condition is satisfied, we need to 
find an optimal dual solution so that we can use Theorem 2(iii). For simplicity suppose that the 
/o; 1 < i < A;, are strictly concave, and also suppose that (5.29) has an optimal solution for every 
A > 0. Then by Exercise 8 of Section 1, for each A > there is a unique optimal solution of (5.29), 
say x*(A). Consider the following algorithm. 



5.2. DUALITY THEORY 67 



Step 1. Select A > arbitrary. Set p = 0, and go to Step 2. 

Step 2. Solve (5.29) for A = A p and obtain the optimal solution x p = (x 1 (\p) , . . . , x k (\p)) . 

k 

Compute eP = Y^ .f OX^)) ~ b. If e p > 0, x p is feasible for (5.28) and can easily be seen to be 

i=l 

optimal. 

Step 3. Set A p=1 according to 

a p+i _ f A? ifef >0 

Ai ~\ \\-dP<?. ifef <0 

where d p > is chosen a priori. Set p = p + 1 and return to Step 3. 

It can be shown that if the step sizes d p are chosen properly, x p will converge to the optimum 
solution of (5.28). For more detail see (Arrow and Hurwicz [I960]), and for other decentralization 
schemes for solving (5.28) see (Geoffrion [1970b]). 



Control of water quality in a stream. 

The discussion in this section is mainly based on (Kendrick, et ai, [1971]). For an informal discus- 
sion of schemes of pollution control which derive their effectiveness from duality theory see (Solow 
[1971]). See (Dorfman and Jacoby [1970].) 

Figure 5.7 is a schematic diagram of a part of a stream into which n sources (industries and 
municipalities) discharge polluting effluents. The pollutants consist of various materials, but for 
simplicity of exposition we assume that their impact on the quality of the stream is measured in 
terms of a single quantity, namely the biochemical oxygen demand (BOD) which they place on the 
dissolved oxygen (DO) in the stream. Since the DO in the stream is used to breakdown chemically 
the pollutants into harmless substances, the quality of the stream improves with the amount of 
DO and decreases with increasing BOD. It is a well-advertized fact that if the DO drops below a 
certain concentration, then life in the stream is seriously threatened; indeed, the stream can "die." 
Therefore, it is important to treat the effluents before they enter the stream in order to reduce the 
BOD to concentration levels which can be safely absorbed by the DO in the stream. In this example 
we are concerned with finding the optimal balance between costs of waste treatment and costs of 
high BOD in the stream. 

We first derive the equations which govern the evolution in time of BOD and DO in the n areas 
of the streams. The fluctuations of BOD and DO will be cyclical with a period of 24 hours. Hence, 
it is enough to study the problem over a 24-hour period. We divide this period into T intervals, 
t = 1, . . . , T. During interval t and in area i let 
Zi(t) = concentration of BOD measured in mg/liter, 
qi(t) = concentration of DO measured in mg/liter, 
Si(t) = concentration of BOD of effluent discharge in mg/liter, and 
raj (i) = amount of effluent discharge in liters. 

The principle of conservation of mass gives us equations (5.31) and (5.32): 

Zi (t + 1) - zi(t) = -a iZi (t) + ^-ig-ift) - *|S + , (5.31) 



qi (t+l)- qi (t) = Pi(q; 
+aiZi(t) - rjiVi, 



t = l,...,Tandz = 1,...,N. 



«(*)) + 



(5.32) 
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Figure 5.7: Schematic of stream with effluent discharges. 



Here, Uj = volume of water in area i measured in liters, ipi = volume of water which flows from 
area i to are i + 1 in each period measured in liters. a.{ is the rate of decay of BOD per interval. This 
decay occurs by combination of BOD and DO. fa is the rate of generation of DO. The increase in 
DO is due to various natural oxygen-producing biochemical reactions in the stream and the increase 
is proportional to (q s — qi) where q s is the saturation level of DO in the stream. Finally, r\i is the 
DO requirement in the bottom sludge. The Vi,ipi,ai,rji,q s are parameters of the stream and are 
assumed known. They may vary with the time interval t. Also zo(t), qo(t) which are the concen- 
trations immediately upstream from area 1 are assumed known. Finally, the initial concentrations 
Zi(l),qi(l), i = 1, . . . , N are assumed known. 

Now suppose that the waste treatment facility in area i removes in interval t a fraction 7Tj(t) of 
the concentration Sj(i) of BOD. Then (5.31) is replaced by 



Zi(t + 1) - Zi{t) = -OLiZi{t) + 



IpiZjjt) , (l-TTi(t))Si (t)mj{t) 
Vi Vi 



(5.33) 



We now turn to the costs associated with waste treatment and pollution. The cost of waste treat- 
ment can be readily identified. In period t the ith facility treats rrii(t) liters of effluent with a BOD 
concentration si{t) mg/liter of which the facility removes a fraction ni(t). Hence, the cost in period 
t will be fi(iri(t),Si(t),m,i(t)) where the function must be monotonically increasing in all of its 
arguments. We further assume that / is convex. 

The costs associated with increased amounts of BOD and reduced amounts of DO are much 
more difficult to quantify since the stream is used by many institutions for a variety of purposes 
(e.g., agricultural, industrial, municipal, recreational), and the disutility caused by a decrease in 
the water quality varies with the user. Therefore, instead of attempting to quantify these costs let 
us suppose that some minimum water quality standards are set. Let q be the minimum acceptable 
DO concentration and let z be the maximum permissible BOD concentration. Then we face the 
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following NP: 

N T 

Maximize - ^ ^ /* (i) , (t) , (t) ) 

i=l t=i 

subject to (5.32), (5.33), and (5 34) 

-qi(t) <-q,i = l,...,N;t = l,...,T, 
Zi(t)<z, i=l,...,N; t = l,...,T, 
< TTi(t) < 1 , i = l,...,N; t = 1, . . . , T. 

Suppose that all the treatment facilities are in the control of a single public agency. Then assuming 
that the agency is required to maintain the standards (q, z) and it does this at a minimum cost it will 
solve the NP (5.34) and arrive at an optimal solution. Let the minimum cost be m(q, z). But if 
there is no such centralized agency, then the individual polluters may not (and usually do not) have 
any incentive to cooperate among themselves to achieve these standards. Furthermore, it does not 
make sense to enforce legally a minimum standard qi{t) > q, zi(t) < z on every polluter since the 
pollution levels in the ith area depend upon the pollution levels on all the other areas lying upstream. 
On the other hand, it may be economically and politically acceptable to tax individual polluters in 
proportion to the amount of pollutants discharged by the individual. The question we now pose 
is whether there exist tax rates such that if each individual polluter minimizes its own total cost 
{i.e., cost of waste treatment + tax on remaining pollutants), then the resulting water quality will be 
acceptable and, furthermore, the resulting amount of waste treatment is carried out at the minimum 
expenditure of resources (i.e., will be an optimal solution of (5.34)). 

It should be clear from the duality theory that the answer is in the affirmative. To see this let 
Wi(t) = (zi(t), —qi(t))', let w(t) = (wi(t), . . . , WN(t)), and let w = (w(l), . . . , w(t)). Then we 
can solve (5.32) and (5.33) for w and obtain 

w = b + Ar , (5.35) 

where the matrix A and the vector b depend upon the known parameters and initial conditions, and 
r is the NT-dimensional vector with components (1 — iti{t))si{t)mi{t). Note that the coefficients 
of the matrix must be non-negative because an increase in any component of r cannot decrease the 
BOD levels and cannot increase the DO levels. Using (5.35) we can rewrite (5.34) as follows: 

Maximize - ^ ^ fi (*) > s i (*) > m i (*)) 

subject to b + Ar < w , (5.36) 
0<7Ti(*)<l, i = 1,...,N; t = 1,...,T, 

where the 2iVT-dimensional vector w has its components equal to —q or z in the obvious manner. 
By the duality theorem there exists a 2iVT-dimensional vector A* > 0, and an optimal solution 
7r*(i), i = 1, . . . , N, t = 1, . . . , T, of the problem: 

Maximize - fi(^i(t), m;(i)) - X*'(b + Ar - w) 

i t (5.37) 
subject to < 7Tj(t) < 1, i = l,...,N; t = 1, . . . , T , 



such that {vr*(t)} is also an optimal solution of (5.36) and, furthermore, the optimal values of (5.36) 
and (5.37) are equal. If we let p* = A'X* > 0, and we write the components of p* as p*(t) to match 
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with the components (1 — ni(t))si(t)rrii(t) of r we can see that (5.37) is equivalent to the set of NT 
problems: 

Maximize - /ifa(i), m^i)) -p*(t)(l - vr i (t))s i (t)m i (t) 

< 7Ti(t) < 1 , (5.38) 

1 = l,. ..,#;* = 1, ...,T. 

Thus, is optimum tax per mg of BOD in area i during period t. 

Before we leave this example let us note that the optimum dual variable or shadow price A* 
plays an important role in a larger framework. We noted earlier that the quality standard (q, z) 
was somewhat arbitrary. Now suppose it is proposed to change the standard in the ith area during 
period t to q + Aqi(t) and z + Azi(t). If the corresponding components of A* are Af (t) and Xf*(t), 
then the change in the minimum cost necessary to achieve the new standard will be approximately 
Xf*(t)Aqi(t) + Xf*(t)Azi(t). This estimate can now serve as a basis in making a benefits/cost 
analysis of the proposed new standard. 



5.3 Quadratic Programming 

An important special case of NP is the quadratic programming (QP) problem: 

Maximize c'x — hx'Px 

(5 39) 

subject to Ax <b, x > , 

where x G R n is the decision variable , c G R n , b G R m are fixed, A is a fixed m x n matrix and 
P = P' is a fixed positive semi-definite matrix. 

Theorem 1: A vector x* G R n is optimal for (5.39) iff there exist A* G R m , n* G R n , such that 

Ax* < b, x* > 

c — Px* = A'X* — /j,*, A* > 0, /x* > , (5.40) 
(X*)'(Ax* - b) = , (>*)'z* = . 

Proof: By Lemma 3 of 1.3, CQ is satisfied, hence the necessity of these conditions follows from 
Theorem 2 of 1.2. On the other hand, since P is positive semi-definite it follows from Exercise 6 
of Section 1.2 that f^-.x^ c'x — 1/2 i'Px is a concave function, so that the sufficiency of these 
conditions follows from Theorem 4 of 1.2. 
From (5.40) we can see that x* is optimal for (5.39) iff there is a solution (x* , y* , X* , [/,* ) to 
(5.41), (5.42), and (5.43): 

Ax + I m Y = b 

-Px-A'X + I n fi = -c, K ' 

x>0y>0, A>0, ^>0, (5.42) 

H'x = , X'y = . (5.43) 

Suppose we try to solve (5.41) and (5.42) by Phase I of the Simplex algorithm (see 4.3.2). Then we 
must apply Phase II to the LP: 

m n 

Maximize — Zj — £j 

i=l j=l 
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subject to 

Ax + I m y + z = b 

-Px - A'X + I n ii + £ = -c 
;c>0,y > 0, A > 0, > 0, z > 0, £ > 



(5.44) 



starting with a basic feasible solution z = b, £ = — c. (We have assumed, without loss of generality, 
that b > and — c > 0.) If (5.41) and (5.42) have a solution then the maximum value in (5.44) is 0. 
We have the following result. 

Lemma 1: If (5.41), (5.42), and (5.43) have a solution, then there is an optimal basic feasible solution 
of (5.44) which is also a solution f (5.41), (5.42), and (5.43). 

Proof: Let x,y,\,fl be a solution of (5.41), (5.42), and (5.43). Then x,y, \,fi,z = 0,£ = is 
an optimal solution of (5.44). Furthermore, from (5.42) and (5.43) we see that at most (n + m) 
components of (x, y, A, p) are non-zero. But then a repetition of the proof of Lemma 1 of 4.3.1 will 
also prove this lemma. 

This lemma suggests that we can apply the Simplex algorithm of 4.3.2 to solve (5.44), starting 
with the basic feasible solution z = b, £ = — c, in order to obtain a solution of (5.41), (5.42), and 
(5.43). However, Step 2 of the Simplex algorithm must be modified as follows to satisfy (5.43): 

If a variable Xj is currently in the basis, do not consider fij as a candidate for entry into the basis; 
if a variable yi is currently in the basis, do not consider \ as a candidate for entry into the basis. If 
it not possible to remove the zi and £j from the basis, stop. 

The above algorithm is due to Wolfe [1959]. The behavior of the algorithm is summarized below. 
Theorem 2: Suppose P is positive definite. The algorithm will stop in a finite number of steps at an 
optimal basic feasible solution (x, y, A, jl, z, £) of (5.44). If z = and | = then (x, y, A, jl) solve 
(5.41), (5.42), and (5.43) and x is an optimal solution of (5.39). If z ^ or £ / 0, then there is no 
solution to (5.41), (5.42), (5.43), and there is no feasible solution of (5.39). 

For a proof of this result as well as for a generalization of the algorithm which permits positive 
semj-definite P see (Cannon, Cullum, and Polak [1970], p. 159 ff). 

5.4 Computational Method 

We return to the general NP (5.45), 



where x E R n , fa : R n — ► R, < i < m, are differentiable. Let O, C R n denote the set of 
feasible solutions. For fell define the function ip(x) : R n — ► it! by 



Maximize fo(x) 
subject to fi(x) < 0, i 



= 1, . . . ,m 



(5.45) 



ma.x{-f 0x (x)h, fi(x) + fi x {x)h, . 



■ ,fm{x) + fmx(x)h}. 



Consider the problem: 



Minimize ip(x)(h) 



subject to — ij){x)(h) — fo x (x)h < , 
-i>(x)(h) + fi(x)f ix h <0 
1 < % < m , -l<hj<l 



1 < j < n ■ 



(5.46) 
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Figure 5.8: h(x k ) is a feasible direction. 



Call h(x) an optimum solution of (5.46) and let ho(x) = ip(x)(h(x)) be the minimum value at- 
tained. (Note that by Exercise 1 of 4.5.1 (5.46) can be solved as an LP.) 

The following algorithm is due to Topkis and Veinott [1967]. 
Step 1. Find x° G $7, set k = 0, and go to Step 2. 

Step 2. Solve (5.46) for x = x k and obtain ho(x k ), h(x k ). If ho(x k ) = 0, stop, otherwise go to Step 
3. 

Step 3. Compute an optimum solution fi(x k ) to the one-dimensional problem, 

Maximize fo(x k + /j,h(x k )) , 

subject to (x k + fih(x k )) £ 1!, |i > , 

and go to Step 4. 

Step 4. Set x k+1 = x k + p,(x k )h(x k ), set k = k + 1 and return to Step 2. 

The performance of the algorithm is summarized below. 
Theorem 1: Suppose that the set 

n(z°) = {x\x e n, /oO) > /o(x )} 

is compact, and has a non-empty interior, which is dense in tt(x°). Let x* be any limit point of 
the sequence x°,x l , . . . ,x k , . . . , generated by the algorithm. Then the Kuhn-Tucker conditions are 
satisfied at x* . 

For a proof of this result and for more efficient algorithms the reader is referred to (Polak [197 1]). 
Remark: If ho(x k ) < in Step 2, then the direction h{x k ) satisfies fo x {x k )h(x k ) > 0, and fi(x k ) + 
f ix (x K )h(x k ) < 0, 1 < i < m. For this reason h(x k ) is called a (desirable) feasible direction. 
(See Figure 5.8.) 
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5.5 Appendix 

The proofs of Lemmas 4,7 of Section 2 are based on the following extremely important theorem 
(see Rockafeller [1970]). 

Separation theorem for convex sets. Let F, G be convex subsets of R n such that the relative interiors 
of F, G are disjoint. Then there exists A G R n , A / 0, and 9 G R such that 

A's < 9 for all geG 
A'/ > 9 for all / G F . 

Proof of Lemma 4: Since M is stable at 6 there exists if such that 

Af (6) - Af (S) < K\b - b\ for all b G 5 . (5.47) 

In i? 1+m consider the sets 

F = {(r, 6)|6 e R m , r> K\b-b\} , 
G = {(r, b)\b G F, r < M{b) - M(S)} . 

It is easy to check that F, G are convex, and (5.47) implies that F n G = <p. Hence, there exist 
(Aq, • • • , A m ) / 0, and 6 such that 



A r + J2 x ibi < for (r, 6) G G , 

i=l 
m 

X r + ^ XA > 6 for (r, b) G F . 



(5.48) 



i=l 

From the definition of F, and the fact that (Ao, ■ ■ ■ , A m ) / 0, it can be verified that (5.49) can hold 

m m 

only if Aq > 0. Also from (5.49) we can see that Aj6; > 8, whereas from (5.48) X fii < 9, 



i=l i=l 



so that Aj6j = 0. But then from (5.48) we get 



i=i 



m (6) - M(h) <^- [e-J2 X M = D- f )(61 " 

t=i j=i 



Proof of Lemma 7: Since 6 is in the interior of B, there exists e > such that 

6 G B whenever \b-b\ < e . (5.49) 

In R 1+m consider the sets 

F = {(r, b)\r > M(b} 

G = {{r,b)\b G B, r < M(b)} . 

Evidently, F, G are convex and F n G = <fi, so that there exist (Ao, ■ ■ ■ , A m ) / 0, and 9 such that 

m 

A r + x i b i > e > for r > M(S) , (5.50) 
i=i 
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A r + A A < ^ , for (r, b) € G . (5.51) 

i=l 

From (5.49), and the fact that (Ao, • • • , A m ) / we can see that (5.50) and (5.51) imply Ao > 0. 
From(5.50),(5.51) we get 



\ M(h) + kk = o 



so that (5.52) implies 

m , 



Chapter 6 



DECISION PROBLEMS: 
DISCRETE-TIME OPTIMAL 
CONTROL 

In this chapter we apply the results of the last two chapters to situations where decisions have to be 
made sequentially over time. A very important class of problems where such situations arise is in 
the control of dynamical systems. In the first section we give two examples, and in Section 2 we 
derive the main result. 



6.1 Examples 

The trajectory of a vertical sounding rocket is controlled by adjusting the rate of fuel ejection which 
generates the thrust force. Specifically suppose that the equations of motion are given by (6.1). 

±i(t) = x 2 (t) 

x 2 (t) = -^p( Xl (t))x 2 2 (t) -g+ £fcu(t) (6.1) 
x 3 (t) = -u(t) , 

where x\{t) is the height of the rocket from the ground at time t, x 2 {t) is the (vertical) speed at 
time t, x%(t) is the weight of the rocket (= weight of remaining fuel) at time t. The "dot" denotes 
differentiation with respect to t. These equations can be derived from the force equations under the 
assumption that there are four forces acting on the rocket, namely: inertia = x%x\ = X3X2; drag 
force = Cdp{x\)x\ where Cd is constant, p{x\) is a friction coefficient depending on atmospheric 
density which is a function of x\\ gravitational force = gx% with g assumed constant; and thrust 
force CtX3, assumed proportional to rate of fuel ejection. See Figure 6.1. The decision variable at 
time t is u(t), the rate of fuel ejection. At time we assume that (xi(0), #2(0), £3(0)) = (0, 0, M); 
that is, the rocket is on the ground, at rest, with initial fuel of weight M. At a prescribed final time 
tf, it is desired that the rocket be at a position as high above the ground as possible. Thus, the 
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decision problem can be formalized as (6.2). 



Maximize x\{tf) 

subject to x(t) = f(x(t),u(t)), < t < tf 
x(0) = (0,0, M) 
u(t) > 0, x 3 (t) > 0, < t < t f 



(6.2) 



where x = (x±, X2, x^)', f '■ R 3+1 — > R 3 is the right-hand side of (6.1). The constraint inequalities 
u(t) > and x$(t) > are obvious physical constraints. 



X3X1 = inertia 



C D Lp[xx)x\ = drag 



CrXz = thrust 




9x3 = gravitational force 



Figure 6. 1 : Forces acting on the rocket. 

The decision problem (6.2) differs from those considered so far in that the decision variables, 
which are functions u : [0,tf] — ► R, cannot be represented as vectors in a yim'te-dimensional 
space. We shall treat such problems in great generality in the succeeding chapters. For the moment 
we assume that for computational or practical reasons it is necessary to approximate or restrict 
the permissible function u(-) to be constant over the intervals [0, t±), [ti,*2)) • • • 1 [tN-i,tf), where 
ti, tit ■ ■ ■ 1 tjv-i are fixed a priori. But then if we let u(i) be the constant value of u(-) over [ti, ii+i), 
we can reformulate (6.2) as (6.3): 

Maximize xi(ijv)(£jv = tf) 

subject to x(t i+1 ) = g(%x(ti),u(i)), i = 0, 1, . . . , N - 1 

x(t ) = x(0) = (0,0,M) ^ 
u(i) >0, x 3 (U) >0, i = 0,l,...,JV . 

In (6.3) (7(1, it(z)) is the state of the rocket at time ti+i when it is in state x(ti) at time ti and 
u(t) = u(i) for ti < t < ti + i. 

As another example consider a simple inventory problem where time enters discretely in a natural 
fashion. The Squeezme Toothpaste Company wants to plan its production and inventory schedule 
for the coming month. It is assumed that the demand on the ith day, < z < 30, is d\{i) for 
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their orange brand and d^ii) for their green brand. To meet unexpected demand it is necessary that 
the inventory stock of either brand should not fall below s > 0. If we let s(i) = (si(i),S2(i))' 
denote the stock at the beginning of the ith day, and m{i) = (mi (i), 7772(7))' denote the amounts 
manufactured on the ith day, then clearly 

s(i + 1) + s(i) + m(i) - d(i) , 

where d{i) = (d\(i), d% (i))'. Suppose that the initial stock is s, and the cost of storing inventory s 
for one day is c(s) whereas the cost of manufacturing amount m is b(m). The the cost-minimization 
decision problem can be formalized as (6.4): 

30 

Maximize y~^(c(g(i)) + 6(777(7))) 

i=o ,, 
subject to s(i + 1) = s(i) + m(i) - d(i), < i < 29 ( bA > 

s(0) = s 

s(i) > {s,s)', m{i) > 0, < i < 30 . 

Before we formulate the general problem let us note that (6.3) and (6.4) are in the form of non- 
linear programming problems. The reason for treating these problems separately is because of their 
practical importance, and because the conditions of optimality take on a special form. 



6.2 Main Result 

The general problem we consider is of the form (6.5). 

N-l 

Maximize /p(i, x(i), u(i)) 
subject to 

dynamics : x(i + 1) — x(i) = f(i,x(i),u(i)), i = 0, . . . , N — 1 , (g_5) 
initial condition: qo(x(0) < 0, go(x(0)) = , 
final condition: q]y(x(N)) < 0, giy(x(N)) = , 
state-space constraint: qi{x{i)) < 0, i = 1, . . . , N — 1 , 
control constraint: hi{u{i)) < 0, i = 0, . . . , N — 1 . 

Herex(i) G u(t) G / (i, v ) : R n+P -> ^, /(»,-,•) : -> g< : ^ n -> 

i? mi , ft : i? n — > -R £i , /ij : i? p —> R Si are given differentiable functions. We follow the control 
theory terminology, and refer to x(i) as the state of the system at time i, and u(i) as the control or 
input at time 7. 

We use the formulation mentioned in the Remark following Theorem 3 of V.1.2, and construct 
the Lagrangian function L by 

L(x(0), x(N);u(0), ...,u(N- l);p(l), . . . ,p(N); 
A ,..., A^; a°, a N ■ 7 ^) 
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N-1 (N-1 



= /o(i, - \ £>(« + l))'(x(i + 1) - s(t) - 

i=0 I i=0 

^(V/ftCxC*)) + (a°)'<7o(*(0)) + (a w ) V(x(^V)) + £ (Y)' W*)) ■ 



i=0 i=0 J 

Suppose that CQ is satisfied for (6.5), and x*(0), . . . , x*(N); u*(0), . . . ,u*(N — 1), is an optimal 
solution. Then by Theorem 2 of 5.1.2, there exist p*(i) in # n for 1 < i < N, A** > in i? m * for 
< i < N, a 1 * in Bh for i = 0,N, and 7** > in R Si for < i < N - 1, such that 

(A) the derivative of L evaluated at these points vanishes, 
and 

(B) AS;(z*00) = for < i < N, f*hi{u*(i)) = 0for0<z<A r -l. 
We explore condition (A) by taking various partial derivatives. 

Differentiating L with respect to x(0) gives 

f 0x (0,x*(0),u*(0)) - {-(p*(l))' - (p*(l)Y[f x (0,x*(0),u*(0))} 
+(A°*)W**(0))] + (a°-)W**(0))]} = , 

or 



p*(0)-p*(l) = [f x (0,x*(0),u*(x))}'p*(l) 
+ [f 0x (0,x*(0),u*(0))}' - [q Qx (x*(0))Y\ *, 



p*(i)-p*(i + 1) = [f x (i,x*(i),u* ({])]' p*(i + 1) 
+ [fox(i,x*(i),u*(i))]' -[qi*(x* (%))]' \* . 



(6.6) 



where we have defined 

P*(0) = [gox(x*(x))Y<*°* ■ (6-7) 
Differentiating L with respect to x(i), 1 < i < N — 1, and re-arranging terms gives 



(6.8) 



Differentiating L with respect to x(iV) gives, 

p*(N) = -[g Nx (x*(N))Ya N * - [q Nx (x*(N))Y\ N * . 
It is convenient to replace a N * by —a N * so that the equation above becomes (6.9) 

P*(N) = [g Nx (x*(N))Ya N * - [q Nx (x* (N))]' \ N > . (6.9) 
Differentiating L with respect to u(i), < i < N — 1 gives 

[/o«(i,x*(i),«*(i))]' + [/«(<,x*(i),«*(i))]y(i + - [M«*(*))]V' = ■ ( 6 - 10 ) 

We summarize our results in a convenient form in 
Table 6.1 



Remark 1: Considerable elegance and mnemonic simplification is achieved if we define the 
Hamiltonian function Hby 



Suppose x*(0), . . . ,x*(N); 


then there exist p*(N); A * , . . . , A ; a * , a N * ' 




u*(Q), . . . , u*(N — 1) maximizes 


7°*, . . . ,7 JV_1 *, such that 




^2 fo(i,x(i),u(i)) subject 

i=0 

to the constraints helow 










dynamics: i = 0, . . . , N — 1 


adjoint equations: i = 0, . . . , N — 1 




x(i + 1) — x(i) = f(i. x(i). u(i)) 


V*U) - V*(i+ I) = \fJi,x*(i),u*(i)Yp*(i + 1) 
+ [fox(i,x*(i),u*(i)]' - [q lx (x*(i)}'^ 




initial condition: 


transversality conditions: 


A * > 0, 


q (x*(0))<0,g (x*(0))=0 


P*(0) = [<? *(^(0))]'a°* 


(A°«)'gb(x*(0)) = 






A^* > 0, 


final conditions: 




qN(x*(N))<0,g N (x*(N))=0 


P*(JV) = [g Nx (x*(N))]'a N * - [q Nx (x* (N))}> \ N * 


(A Ar *)'^(x*(iV))=0 


state space constraint: 






i = l,...,N -1 




A** > 0, 


Qi(x*(i)) < 




(\ i *y qi (x*(i)) = o 


control constraint: 






i = 0,...,N -1 


[f 0u (i,x*(i),u* (i))]' + [/ w (i,x*(*)u*(z))]'. 


7 J * > 


hi{u*{i)) < 


P*(ti) = [M«*«)]V* 


(7 i *)'/i i (n*(i) = 
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H(i,x,u,p) = f (i,x,u) + p'f(i,x,u) . 
The dynamic equations then become 

x*(i + 1) -x*(i) = [H p (i,x*(i),u*(i),p*(i + 1))]' 



0<i<N-l. (6 " U) 



and the adjoint equations (6.6) and (6.8) become 



P*(i)-P*(i + 1) = [H x (i,x*(i),u*(i),u*(i),p*(i + 1))]' - [q ix (x* ({})}' X^, 

< * < N - 1 , 

whereas (6.10) becomes 

[Mu*(*))]V* = [H u (i,x*(i),u*(i),p*(i + 1))}', 0<i<JV-l. (6.12) 
Remark 2: If we linearize the dynamic equations about the optimal solution we obtain 

Sx(i + 1) - Sx(i) = [f x (i,x*(i),u*(i))]5x(i) + [f u (i, x*(i, x*, (i),u*(i))]6u(i) , 
whose homogeneous part is 

z(i + i)-z(i) = \f x (i,x*(i),u*(i))]z(i), 

which has for it adjoint the system 

r(i) - r(i + 1) = [/ x (i, «*(i))]'r(i + 1) . (6.13) 

Since the homogeneous part of the linear difference equations (6.6), (6.8) is (6.13), we call (6.6), 
(6.8) the adjoint equations, and the p*{i) are called adjoint variables. 

Remark 3: If the fo(i, •, •) are concave and the remaining function in (6.5) are linear, then CQ is 
satisfied, and the necessary conditions of Table 6. 1 are also sufficient. Furthermore, in this case we 
see from (6.13) that u*(i) is an optimal solution of 

Maximize H(i,x*(i),u,p*(i + 1)), 
subject to hi(u) < . 

For this reason the result is sometimes called the maximum principle. 

Remark 4: The conditions (6.7), (6.9) are called transversality conditions for the following reason. 
Suppose qo = 0, qN = 0, so that the initial and final conditions read go(x(0)) = 0, gN(x(N)) = 0, 
which describe surfaces in R n . Conditions (6.7), (6.9) become respectively p*(0) = [gox(x*(0))]'oP* , p*(N) = 
[gNx(x(N))]'a N * which means that p*(0) and p*(N) are respectively orthogonal or transversal to 
the initial and final surfaces. Furthermore, we note that in this case the initial and final conditions 
specify (Iq + £ n ) conditions whereas the transversality conditions specify (n — £q) + (n — £ n ) condi- 
tions. Thus, we have a total of 2n boundary conditions for the 2n-dimensional system of difference 
equations (6.5), (6.12); but note that these In boundary conditions are mixed, i.e., some of them 
refer to the initial time and the rest refer to the final time. 
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Exercise 1: For the regulator problem, 

N-l „ N-l 



Maximize | x(i)'Qx(i) + - u(i)'Pu(i 



2 

i=0 i=0 

subject to x(i + 1) - = Ax(i) + Bu(i), < i < N - 1 
x(0) = x(0), 

€ R p , < t < JV - 1 , 

where G yl and B are constant matrices, x(0) is fixed, Q = Q' is positive semi-definite, 
and P = P' is positive definite, show that the optimal solution is unique and can be obtained by 
solving a 2n-dimensional linear difference equation with mixed boundary conditions. 

Exercise 2: Show that the minimal fuel problem, 

N-l / P 

Minimize 

i=0 \j=l 

subject to x(i + 1) - = + fiu(i), < i < N - 1 

s(0) = x(0), x(N) = x(N) , 
u(i) e P p , |«(i))i| < 1, 1 < J < p, < i < N - 1 

can be transformed into a linear programming problem. Here x(0),x(N) are fixed, A and B are as 
in Exercise 1 . 
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Chapter 7 



SEQUENTIAL DECISION PROBLEMS: 

CONTINUOUS-TIME OPTIMAL 
CONTROL OF LINEAR SYSTEMS 



We will investigate decision problems similar to those studied in the last chapter with one (math- 
ematically) crucial difference. A choice of control has to be made at each instant of time t where 
t varies continuously over a finite interval. The evolution in time of the state of the systems to be 
controlled is governed by a differential equation of the form: 



where x(t) G R n and u(t) G R p are respectively the state and control of the system at time t. 

To understand the main ideas and techniques of analysis it will prove profitable to study the linear 
case first. The general nonlinear case is deferred to the next chapter. In Section 1 we present the 
general linear problem and study the case where the initial and final conditions are particularly 
simple. In Section 2 we study more general boundary conditions. 

7.1 The Linear Optimal Control Problem 

We consider a dynamical system governed by the linear differential equation (7.1): 



Here A(-) and B(-) are n x n- and n x p-matrix valued functions of time; we assume that they are 
piecewise continuous functions. The control u(-) is constrained to take values in a fixed set f2 C BP, 
and to be piecewise continuous. 

Definition: A piecewise continuous function u : [to, oo) — > will be called an admissible control. 
U denotes the set of all admissible controls. 

Let c G R n , x° G R n be fixed and let tj > to be a fixed time. We are concerned with the 




x(t) = f{t,x{t),u(t)) 



x(t) = A(t)x(t) + B(t)u(t), t>t . 



(7.1) 
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decision problem (7.2). 



Maximize c'x(t/), 
subject to 

dynamics: x(t) = A(t)x(t) + B(t)u(t) , to <t <tf , 
initial condition: a; (to) = %° > 
final condition: x(t/) G R n , 
control constraint: u(-) G U . 



Definition: (i) For any piecewise continuous function u(-) : [to,t/] R p > f° r an y z £ R n > and 
any to < t± < f 2 < t/ let 



denote the state of (7.1) at time t2, if a time t± it is in state z, and the control u(-) is applied, 
(ii) Let 



Thus, K{t2,t\,z) is the set of states reachable at time t2 starting at time t\ in state z and using 
admissible controls. We call K the reachable set. 

Definition: Let <I>(t, r), to < r < t < tj, be the transition-matrix function of the homogeneous 
part of (7.1), i.e., $ satisfies the differential equation 



Exercise 1: (i) Assuming that 0, is convex, show that U is a convex set. (ii) Assuming that U is 
convex show that Kfa, ti, is a convex set. (It is a deep result that K(t2,t\, z) is convex even if 
Q is not convex (see Neustadt [1963]), provided we include in U any measurable function 

u : [to, oo) — > O.) 

Definition: Let A' C i? rt , and let x* £ We say that c is the outward normal to a hyperplane 
supporting K at x* if c 7^ 0, and 

c'x* > c'x for all x G if . 

The next result gives a geometric characterization of the optimal solutions of (2). 
Lemma 2: Suppose c / 0. Let u*{-) G and let x*(t) = c/)(t,to,x ,u*). Then n* is an optimal 
solution of (2) iff 

(i) x*(tf) is on the boundary of K = K(tj, to, x°), and 

(ii) c is the outward normal to a hyperplane supporting K at x*. (See Figure 7.1.) 

Proof: Clearly (i) is implied by (ii) because if x*(t/) is in the interior of K there is 5 > such 
that (x*(t f ) + 5c) G K; but then 



0(t 2 ,ti,<z,tt) 



K(t 2 , ti, 2:) = {0(t 2 , ti, z, u)\u G £/} . 
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Xl 

Figure 7.1: c is the outward normal to ir* supporting K at x*(tf) 



c'{x*(t f ) + 5c) = c'x*(t f ) + <5|c| 2 > c'x*(t f ) . 

Finally, from the definition of K it follows immediately that u* is optimal iff c'x*(tf) > c'x for all 
x E K . 
The result above characterizes the optimal control u* in terms of the final state x*(tj). The beauty 
and utility of the theory lies in the following result which translates this characterization directly in 
terms of u* . 

Theorem 1: Let u*(-) £ U and let x*{t) = (p(t,to, x° ,u*), t < t < tf. Let p*(t) be the solution 
of (7.3) and (7.4): 

adjoint equation: p*(t) = —A'(t)p*(t) , t <t <tf. (7.3) 

final condition: p*{tf) = c . (7.4) 
Then u*(-) is optimal iff 

(p*(t))'B(t)u*(t) = sup{(p*(t))'B(t)v\v G n} , (7.5) 
for all t G [to , tf] , except possibly for a finite set. 
Proof: u*(-) is optimal iff for every u(-) G U 



(p*(t f )YMt f ,to)x + f^(t f ,T)B(r)u*(r)dr] 

> ( P *(t f )ymt f ,to)x° + jlf^t f ,T)B(T) U (T)dT) , 

which is equivalent to (7.6). 

>&{p*{t f ))'*{t f ,T)B{T)u{T)dT 
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Now by properties of the adjoint equation we know that p*(t))' = (p*(tf))'3>(tf,t) so that (7.6) is 
equivalent to (7.7), 

^{p*{T))'B{T)u*{T)dT > ^(p*{T))'B{T)u{T)dT, (7.7) 

and the sufficiency of (7.5) is immediate. 

To prove the necessity let D be the finite set of points where the function B(-) or «*(•) is discon- 
tinuous. We shall show that if «*(•) is optimal then (7.5) is satisfied for t G" D. Indeed if this is not 
the case, then there exists t* G [io,i/]) t* D, and v G such that 

(p*(t*))'B(t*)u*(t*) < {p*{t*))'B{t*)v , 

and since t* is a point of continuity of B(-) and u*(-), it follows that there exists 5 > such that 

(p*(t))'B(t)u*{t) < (p*(t))'B(t)v, for \t - t*\ < 5 . (7.8) 

Define u(-) GUby 

r v \t-t*\ <s, t€[to,t f ] 

^ ' \ u*(t) otherwise . 

Then (7.8) implies that 

f%(p*(t))>B(t)u(t)dt > ^{p*{t))'B{t)u*{t)dt . 
But then from (7.7) we see that «*(•) cannot be optimal, giving a contradiction. 



Corollary 1: For to < t\ < t<z < tf, 

(p*(t 2 ))x*(t 2 ) > {p*(t 2 ))'x for all x G K(h,ti,x*(h)). (7.9) 
Exercise 2: Prove Corollary 1. 

Remark 1: The geometric meaning of (7.9) is the following. Taking t\ = to in (7.9), we see that if 
«*(•) is optimal, i.e., if c = p*(tf) is the outward normal to a hyperplane supporting K(tf,to,x°) 
at x*(tf), then x*(t) is on the boundary of K(t, to, x°) and p*(t) is the normal to a hyperplane 
supporting K(t, to, x°) at x*(t). This normal is obtained by transporting backwards in time, via 
the adjoint differential equation, the outward normal p*(tj) at time tf. The situation is illustrated 
in Figure 7.2. 

Remark 2: If we define the Hamiltonian function H by 

H(t,x,u,p) =p'(A(t)x + B(t)u) , 

and we define M by 

M(t,x,p) = sup{H(t,x,u,p)\u G Q}, 

then (7.5) can be rewritten as 

H(t,x*(t),u*(t),p*(t)) =M(t,x*(t),p*(t)) . (7.10) 
This condition is known as the maximum principle. 
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Exercise 3: (i) Show that m(t) = M(t, x*(t),p*(t)) is a Lipschitz function of t. (ii) If A(t), B(t) 
are constant, show that m(t) is constant. (Hint: Show that (dm/dt) = 0.) 

The next two exercises show how we can obtain important qualitative properties of an optimal 
control. 

Exercise 4: Suppose that 12 is bounded and closed. Show that there exists an optimal control «*(•) 
such that u* (t) belongs to the boundary of 12 for all t. 

Exercise 5: Suppose 12 = [a, f5], so that B(t) is an n x 1 matrix. Suppose that A(t) = A and 
B(i) = B are constant matrices and A has n real eigenvalues. Show that there is an optimal 
control and t§ <t\<ti < . . . <t n <tf such that u*(t) = a or j3 on [ij, tj+i), < i < n. 
(Hint: first show that {p*{t))'B = 71 txxy{5\t) + . . . + j n exp((5 n (i)) for some ji, 5{ in R.) 
Exercise 6: Assume that K(tf, to,x°) is convex (see remark in Exercise 1 above). Let 
fo : R n — > R be a differentiable function and suppose that the objective function in (7.2) is 
fo(x(tf)) instead of c'x(tf). Suppose u*(-) is an optimal control. Show that u*(-) satisfies the 
maximum principle (7.10) where is the solution of the adjoint equation (7.3) with the final 
condition 

p*(t f ) = vfo(x*(t f )) . 

Also show that this condition is sufficient for optimality if fo is concave. (Hint: Use Lemma 1 of 
5.1.1 to show that if u*(-) is optimal, then f 0x (x*(tf)(x*(tf) - x) < for all x G K(tf,t ,x ).) 



7.2 More General Boundary Conditions 

We consider the following generalization of (7.2). The notion of the previous section is retained. 

Maximize c'x(tf) 
subject to 

dynamics: x(t) = A(t)x(t) + B(t)u(t), to <t<tf , 

initial condition: G°x(to)=b°, (7.11) 
final condition: G^x(tj) = b$ , 

control constraint: u(-) £U, i.e., n : [U,, Li A — > ® and 
u(-)piecewise continuous. 

In (7.11) G° and G f are fixed matrices of dimensions £°xn and $ x n respectively, while b° G 
R io , V G R l s are fixed vectors. 

We will analyze the problem in the same way as before. That is, we first characterize optimality 
in terms of the state at the final time, and then translate these conditions in terms of the control. For 
convenience let 

T° = {z G R n \G°z = b } , 
Tf = {2 G R n \G f z = b f } . 

Definition: Let p G R n . Let z* G T . We say that p is orthogonal to T° at z* and we write 

p _L T°{z*) if 
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Figure 7.2: Illustration of (7.9) for t\ = to- 
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p'( z - z *) = for all z G T° . 
Similarly if z* G T- f , p _L T^z*) if 

- z *) = for all z eT- f . 
Definition: LetX(t f ) = {$(*/, t Q )z + w\z G T°, w G if (t/,i o ,0)}. 
Exercise 1: X(t/) = {*(*/, to, z, u)|z G T°, u(-) G W}. 

Lemma 1: Let x*(i ) G T° and u*(-) G W. Let = (t)(t,t ,x*(t ),u*), and suppose that 

G 7^. 

(i) Suppose the 17 is convex. If «*(•) is optimal, there exist po £ ^)Po > and p G R n , not both 
zero, such that 

(p c + p)'x*(t f ) > (p c + p)'x for all x G , (7.12) 

p±Tf(x*(t f )) , (7.13) 



[*(*/, *o)]'(Poc + p) -L T°(s*(t )) • (7.14) 

(ii) Conversely if there exist po > 0, and p such that (7.12) and (7.13) are satisfied, then u*(-) is 
optimal and (7.14) is also satisfied. 
Proof: Clearly «*(•) is optimal iff 

c'x*{t f ) > c'x for all x G X(t f ) n . (7.15) 

(i) Suppose that u*(-) is optimal. In R 1+m define sets S 1 , S 2 by 

5 1 = {{r,x)\r > c'x*(tj),x G T-^} , (7.16) 



5 2 = {(r,x)\r = c'x , x G X(t f )} . (7.17) 

First of all S 1 n S 2 = <f> because otherwise there exists x G X(tf ) n such that c'x > c'x*(tf) 
contradicting optimality of u*(-) by (7.15). 

Secondly, S is convex since is convex. Since is convex by hypothesis it follows by Exercise 
1 of Section 1 that S 2 is convex. 

But then by the separation theorem for convex sets (see 5.5) there exists po G R,p G R n , not 
both zero, such that 

por 1 +p'x l >p r 2 +p'x 2 for all (r^ x 1 ) G S i , i = 1,2. (7.18) 

In particular (7.18) implies that 

p r +p'x*(tf) > pqc'x + p'x for all x G X(tj),r > c'x*(tj). (7.19) 

Letting r — ► oo we conclude that (7.19) can hold only if po > 0. On the other hand letting r — ► 
c'x*(tf) we see that (7.19) can hold only if 

p c'x*(tf)+p'x*(tf) > pqc'x + p'x for all x G X(t/) , (7.20) 

which is the same as (7.12). Also from (7.18) we get 
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p r + pfx > p c'x*(tf) +p'x*(tj) for all r > c'x*(tf),x G T f 



which can hold only if 



Pic'x* (t f )+p'x > p c'x*(t f ) + p'x*(t f ) for all x G T- f , 



or 



p'{x — X 



*{t f )) > for all x eTf 



(7.21) 



But {x - x*(t f )\x G T-f } = {z\G f z = 0} is a subspace of i? n , so that (7.21) can hold only if 

p'(x - x*(t f )) = for all x £ T f , 

which is the same as (7.13). Finally (7.12) always implies (7.14), because by the definition of X(tf) 
and Exercise 1, t )(z - x*(t )) + x*(t f )} G X(t f ) for all z G T°, so that from (7.12) we 

get 

> (p c + p)'$(tf,t )(z - x*(t )) for all z G T° , 
which can hold only if (7.14) holds. 

(ii) Now suppose that po > and p are such that (7.12), (7.13) are satisfied. Let x G X{tt) n . 
Then from (7.13) we conclude that 



Remark 1: If it is possible to choose po > then fto = 1, p = (p/po) will also satisfy (7.12), 
(7.13), and (7.14). In particular, in part (ii) of the Lemma we may assume po = 1. 
Remark 2: it would be natural to conjecture that in part (i) po may be chosen > 0. But in Figure 
7.3 below, we illustrate a 2-dimensional situation where T° = {x }, is the vertical line, and 
T' n X(tf) consists of just one vector. It follows that the control «*(•) G U for which 
x*(tf) = 4>(tf, to,x°,u*) G T* is optimal for any c. Clearly then for some c (in particular for the 
c in Figure 7.3) we are forced to set po = 0. In higher dimensions the reasons may be more 
complicated, but basically if is "tangent" to X(tf) we may be forced to set po = (see 
Exercise 2 below). Finally, we note that part (i) is not too useful if po = 0, since then (7.12), (7.13), 
and (7.14) hold for any vector c whatsoever. Intuitively po = means that it is so difficult to satisfy 
the initial and final boundary conditions in (7.11) that optimization becomes a secondary matter. 
Remark 3: In (i) the convexity of O, is only used to guarantee that K(tt,to, 0) is convex. But it is 
known that K(tf, to, 0) is convex even if O is not (see Neustadt [1963]). 

Exercise 2: Suppose there exists z in the interior of X(tf) such that z G T*. Then in part (i) we 
must have po > 0. 



p'x*(tf) = p'x 



so that from (7.12) we get 



Poc'x*{tf) > p Q dx ; 



but then by (7.15) u*{-) is optimal. 







We now translate the conditions obtained in Lemma 1 in terms of the control u* . 
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Theorem 1: Letx*(t ) G T° ancU*(-) G U. Letx*(i) = </>(t, t , x*(t ), u*) and suppose that 
x*(t/) e Tf. 

(i) Suppose that $7 is convex. If u* (•) is optimal for (7. 11), then there exist a number p* > 0, and a 
function p* : [to, tf] — > -R n , not both identically zero, satisfying 



adjoint equation: p*(t) = —A'(t)p*(t) , to <t<tf (7.22) 

initial condition: p* (to) A.T (x* (to)) (7.23) 

final condition: (p*(t f ) - p f c)±Tf (x*(t f )) , (7.24) 
and the maximum principle 

H(t,x*(t),u*(t),p*(t)) = M(t,x*(t),p*(t)) , (7.25) 
holds for all t E [to, tf] except possibly for a finite set. 



(ii) Conversely suppose there exist p$ > and p*(-) satisfying (7.22), (7.23), (7.24), and (7.25). 
Then u*(-) is optimal. 

[Here 

H(t,x,u,p) = p'{A{t)x + B(t)u), M(t,x,p) = sup{H(t,x,v,p)\v £ Q}.] 

Proof: A repetition of a part of the argument in the proof of Theorem 1 of Section 1 show that if p* 
satisfies (7.22), then (7.25) is equivalent to (7.26): 

(p*(tf))'x*(tf) > (p*(t f )Yx for all x G K(t f ,t ,x*(t Q )) . (7.26) 

(i) Suppose u*(-) is optimal and is convex. Then by Lemma 1 there exist p > 0, p G i? n , not 
both zero, such that (7.12), (7.13) and (7.14) are satisfied. Let p$ = po and letp*(-) be the solution 
of (7.22) with the final condition 

P*(tf) =PqC + p = p c + p . 

Then (7.14) and (7.13) are respectively equivalent to (7.23) and (7.24), whereas since K(tf,to, a;* (to)) C 
X(t f ), (7.26) is implied by (7.12). 

(ii) Suppose p* Q > and (7.22), (7.23), (7.24), and (7.26) are satisfied. Let p = p* G and p = 
p*(tf) — PqC, so that (7.24) becomes equivalent to (7.13). Next if x G X(tf) we have 

(p c + p)'x = (p*(tf))'x 

= (p*(tf))'($(tf,t )z + w) , 
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Figure 7.3: Situation where po = 
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for some z G T° and some w G K(tf, to,0). Hence 

(poc + p)'sc= (p* (/))'*(*/, t )(«-x*(to)) 

+ t )x*(i )) 
= ( P %t )Y(z-x%t )) 
+(p*(t f ))'(w + $(t f ,t )x*(t )) . 

But by (7.23) the first term on the right vanishes, and since (w+(j)(tf,to)x*(to)) G K(tf, to, x*(to)), 
it follows from (7.26) that the second term is bounded by (p*(tf))'x*(tf). Thus 

(p c + p)' x* (t f) > (p c + p)'x for all x G X(tf) , 

and so u*(-) is optimal by Lemma 1. 

Exercise 3: Suppose that the control constraint set is Q(t) which varies continuously with t, and 
we require that u(t) G £l(t) for all t. Show that Theorem 1 also holds for this case where, in (7.25), 
M(t,x,p) =&up{H(t,x,v,p)\v G Q(t)}. 

Exercise 4: How would you use Exercise 3 to solve Example 3 of Chapter 1 ? 
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Chapter 8 

SEQUENTIAL DECISION PROBLEMS: 
CONTINUOUS-TIME OPTIMAL 
CONTROL OF NONLINEAR SYSTEMS 



We now present a sweeping generalization of the problem studied in the last chapter. Unfortunately 
we are forced to omit the proofs of the results since they require a level of mathematical sophis- 
tication beyond the scope of these Notes. However, it is possible to convey the main ideas of the 
proofs at an intuitive level and we shall do so. (For complete proofs see (Lee and Markus [1967] 
or Pontryagin, et al, [1962].) The principal result, which is a direct generalization of Theorem 1 of 
7.2 is presented in Section 1. An alternative form of the objective function is discussed in Section 
2. Section 3 deals with the minimum-time problem and Section 4 considers the important special 
case of linear systems with quadratic cost. Finally, in Section 5 we discuss the so-called singular 
case and also analyze Example 4 of Chapter 1. 



8.1 Main Results 

8.1.1 Preliminary results based on differential equation theory. 

We are interested in the optimal control of a system whose dynamics are governed by the nonlinear 
differential equation 

±{t) = f{t,x,{t),u{t)) , to <*<*/, (8-D 

where x(t) G R n is the state and u(t) G BP is the control. Suppose u*(-) is an optimal control 
and £*(■) is the corresponding trajectory. In the case of linear systems we obtained the necessary 
conditions for optimality by comparing x*{-) with trajectories x(-) corresponding to other admis- 
sible controls u(-). This comparison was possible because we had an explicitly characterization of 
x(-) in terms of u(-). Unfortunately when / is nonlinear such a characterization is not available. 
Instead we shall settle for a comparison between the trajectory £*(•) and trajectories x(-) obtained 
by perturbing the control u*(-) and the initial condition x*(to). We can then estimate the difference 
between x(-) and £*(•) by the solution to a linear differential equation as shown in Lemma 1 below. 
But first we need to impose some regularity conditions on the differential equation (8.1). We assume 
throughout that the function / : [to, tf] x R n xfi p -> R n satisfies the following conditions: 
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1. for each fixed t G [*o, t^], /(t, -, •) : R n xR p — ► i? n is continuously differentiable in the 
remaining variables (sc, it), 

2. except for a finite subset D C [to>*/]> the functions /, f x , f u are continuous on [to, tf] x R n x 

and 

3. for every finite a, there exist finite number (3 and 7 such that 

|/(t, x,u)\ < (3 + j\x\ for all t G [to,*/]) ^ G R n , u £ R p with |«| < a . 

The following result is proved in every standard treatise on differential equations. 

Theorem 1: For every z G R n , for every t\ G [to, tf], and every piecewise continuous function 
u(-) : [to, tf] — > i? p , there exists a unique solution 

x(t) = 0(t,tl, £,«(•)) , ti <t<tf , 

of the differential equation 

x(t) = f(t,x(t),u(t)) ,h<t<t f , 

satisfying the initial condition 

x(t\) = z . 

Furthermore, for fixed t\ < ti in [to, t/] and fixed u{-), the function 4>(t2,ti, •, u(-)) : R n — > R n is 
differentiable. Moreover, the n x n matrix-valued function defined by 

<f>(t 2 ,t 1} z,u(-)) = g£(t 2 ,ti,zX-)) 
is the solution of the linear homogeneous differential equation 

ff(i,ti,z, «,(•)) = [H(t,x,(t),«(t))]$(t,ti,z, «(■)), h<t<tf, 
and the initial condition 

$(tl,tl,2:,lt(-)) = /„ • 

Now let C RP be a fixed set and let be set of all piecewise continuous functions u(-) : 
[to,tf] —> 17. Let u*(-) G U be fixed and let D* be the set of discontinuity points of u*(-). Let 
Xq G i?" be a fixed initial condition. 

Definition: n = (ti, . . . , t m ; €1, . . . , t m \ u±, . . . , u m ) is said to be a perturbation data for u*(-) if 

1. m is a nonnegative integer, 

2. t < ti < t2 < • • • t m < tf, and ti g D*\JD, i = 1, . . . , m (recall that D is the set of 
discontinuity points of /), 

3. li > 0, i = 1, . . . ,m, and 

4. Uj G fi, 2 = 1, . . . , m. 
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Let e(ir) > be such that for < e < e(ir) we have [U — e£i,ti] C [io>*/] f° r a U h an d 
[ti — e£i,ti] f)[tj —e£j,tj] = 4> for i / j. Then for < e < e(n),the perturbed control U( 7r)E )(-) G ^ 
corresponding to ir is defined by 



V,e)(*) 



Uj for all t E [ti — eti , tj] , i = 1, . . . , m 
it* (t) otherwise . 



Definition: Any vector £ G i? n is said to be a perturbation for Xq, and a function £(£ )£ ) denned for 
e > is said to be a perturbed initial condition if 



lim nr.ft \ = r% 

and 

Now let x*(t) = t , ^ ""*(•)) and let z e (i) = i>(t,h,x^,u^ e -)(-)).Let $(* 2 ,*i) = 
3>(t 2 , £i, The following lemma gives an estimate of — x £ (i). The proof of the 

lemma is a straightforward exercise in estimating differences of solutions to differential equations, 
and it is omitted (see for example (Lee and Markus [1967])). 

Lemma 1: lim \x e (t) — x*(t) — eh/^ £ ){t)\ = for t G [io>£tL where h/ n £ )(-) is given by 



Ve)(*) = $ (*'*o)S ,te[t ,h) 

= $(t, t„)£ + *i)[/(*i, «i) - f(tl,X*(h), , 4 G [i!,i 2 ) 

j 

i=i 

(See Figure 8.1.) 

We call ^(7r,$)(*) the linearized (trajectory) perturbation corresponding to (7r,£). 
Definition: For z G i? n , f G [io>*/] let 

K(t,t ,z) = {(/)(t,t ,z,u(-))\u{-) G £Y} 

be the set of states reachable at time t, starting at time to in state z, and using controls u(-) G U. 
Definition: For each t G [to, tf], let 

= {hr n Q\(t)\iris a perturbation data for «*(•), and 
/i(7r,o)(")i s the linearized perturbation 
corresponding to(7r, 0)} . 

Remark: By Lemma 1 (x*(t)+eht 7r ^) belongs to the set K (t, to, aJ(f, e )) up to an error of order o(e). 
In particular for £ = 0, the set x*(t) + can serve as an approximation to the set K(t, to, Xq). 
More precisely we have the following result which we leave as an exercise. 
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Figure 8.1: Illustration for Lemma 1. 



Exercise 1: (Recall the definition of the tangent cone in 5.1.1.) Show that 

Q(t) cC(K(t,t ,x* ),x*(t)) . (8.2) 

We can now prove a generalization of Theorem 1 of 7.1. 
Theorem 2: Consider the optimal control problem (8.3): 



Maximize ip(x(tf)) 
subject to 

dynamics: x(t) = f(t,x(t),u(t)) , to <t < tf , 

initial condition: x(to) = Xq , (8.3) 
final condition: x(tf) E R n , 

control constraint: u{-) £ U, i.e., u : [to,tf] — > $7 and 
u(-) piecewise continuous , 

where ifi : R n — > R is differentiable and / satisfies the conditions listed earlier. 

Let u*(-) £ U be an optimal control and let x*{t) = <f>(t, to, Xq, u*(-)), to < t < tf, be the 
corresponding trajectory. Let p*(t), to < t < tf, be the solution of (8.4) and (8.5): 



adjoint equation: p*(t) = -[^(t,x*(t),u*(t))]'p*(t), t <t <tf , 



(8.4) 
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final condition: p*(tf) = \/tp(x*(tf)) . (8.5) 

Then satisfies the maximum principle 

H(t,x*(t),u*(t),p*(t)) = M(t,x*(t),p*(t)) (8.6) 

for all t G [to,tf] except possibly for a finite set. [Here H(t, x, u,p) = p'f(t, x,u,), M(t, x,p) = 

sup{H(t, x, v,p)\v G fi}]. 

Proof: Since «*(•) is optimal we must have 

i>{x*{tf)) > ^0) for all * G K(t f ,t ,x* ) , 
and so by Lemma 1 of 5. 1. 1 

ifj(x*(t f ))h < for all /i G C(K(t f ,t ,x* ),x*(t f )) , 
and in particular from (8.2) 

ipx(x*(tf))h < for all h G Q(tf) . (8.7) 
Now suppose that (8.6) does not hold from some t* G" D* U D. Then there exists ved such that 

p*(i*)'[/(f,s(O,u)-/(t*,a?(t*),«*(i*))]>0. (8.8) 
If we consider the perturbation data n = (t*;l;v), then (8.8) is equivalent to 

P*(**)Vo)(**)>° . (8.9) 

Now from (8.4) we can see that p*(t*)' = p*(tf)'$(t f ,t*). Also h {7T)0) (t f ) = $(tf,t*)h(n, )(t*) 
so that (8.9) is equivalent to 

P*(tf)'h {7T ,o)(tf) >0 

which contradicts (8.7). 
8.1.2 More general boundary conditions. 

In Theorem 2 the initial condition is fixed and the final condition is free. The problem involving 
more general boundary conditions is much more complicated and requires more refined analysis. 
Specifically, Lemma 1 needs to be extended to Lemma 2 below. But first we need some simple 
properties of the sets Q(t) which we leave as exercises. 

Exercise 2: Show that 

(i) Q(t) is a cone, i.e., if h G Q(t) and A > 0, then Xh G Q{t), 

(ii) for t < h < t 2 < t f , *(t 2 ,ti)Q(*i) C Q{t 2 ) . 
Definition: Let C(t) denote the closure of Q(t). 
Exercise 3: Show that 

(i) C (t) is a convex cone, 

(ii) for t < h < t 2 < t f , *(* 2 ,*i)C(ti) C C(t 2 ) . 

Remark: From Lemma 1 we know that if h G C(t) then (x*(£) +e/i) belongs to K{t, to, x*(to)) up 
to an error of order o(e). Lemma 2, below, asserts further that if h is in the interior of C(t) then in 
fact (x*(t) + eh) G K(t, to, x*(to)) for e > sufficiently small. The proof of the lemma depends 
upon some deep topological results and is omitted. Instead we offer a plausibility argument. 
Lemma 2: Let h belong to the interior of the cone C(t). Then for all e > sufficiently small, 
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(x*(t) + sh) G K{t,t ,x* ) . (8.10) 
Plausibility argument. (8.10) is equivalent to 

eh£K(t,t ,x*(t ))-{x*(t)} , (8.11) 
where we have moved the origin to x*(t). The situation is depicted in Figure 8.2. 




Figure 8.2: Illustration for Lemma 2. 

Let C{e) be the cross-section of C(t) by a plane orthogonal to h and passing through eh. Let 
K{e) be the cross-section of K(t, to, Xq) — {x*(to)} by the same plane. We note the following: 

(i) by Lemma 1 the distance between C(e) and K(e) is of the order o(e); 

(ii) since h is in the interior of C(t), the minimum distance between eh and C{e) is 5e where 
5 > is independent of e. 

Hence for e > sufficiently small eh must be trapped inside the set K(e). 

(This would constitute a proof except that for the argument to work we need to show that there 
are no "holes" in K(e) through which eh can "escape." The complications in a rigorous proof arise 
precisely from this drawback in our plausibility argument.) 

Lemmas 1 and 2 give us a characterization of K(t, to, Xq) in a neighborhood of x*(t) when we 
perturb the control «*(•) leaving the initial condition fixed. Lemma 3 extends Lemma 2 to the case 
when we also allow the initial condition to vary over a fixed surface in a neighborhood of Xq. 

Let g° : R n — > R e ° be a differentiable function such that the £q x n matrix (x) has rank 
£ for all x. Let b° G R n be fixed and let T° = {x\g°(x) - b }. Suppose that x* Q G T° and let 

T°(xq) = {£\g°(xo)€ = °l- Tnus ' ^(^o) + i x o} is tne P lane through x* Q tangent to the surface 

T°. The proof of Lemma 3 is similar to that of Lemma 2 and is omitted also. 

Lemma 3: Let h belong to the interior of the cone {C(t)+$(t, t )T°(xl)}. For e > let h(e) G R n 

be such that lim hie) = 0, and limf-We) = h. Then for e > sufficiently small there exists 

e^0 e 

x (e) G T° such that 



(x*(t) + h{e)) G K(t, to, xq(e)) 
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We can now prove the main result of this chapter. We keep all the notation introduced above. 
Further, let gf : R n — ► R e f be a differentiable function such that g[(x) has rank £f for all x. 
Let b f G R n be fixed and let T- f = {x\g f (x) - b f }. Finally, if x*{t f ) G T f let (x*(t f )) = 

ti\gkx*(t f m = 0}. 

Theorem 3: Consider the optimal control problem (8.12): 

Maximize tp(x(tf)) 
subject to 

dynamics: x(t) = f(t,x(i),u(t)) , to < t <tf , 

initial conditions: g°(x(to))=b° , (8-12) 
final conditions: g*(x(tf))=b* , 
control constraint: u(-) G U,i.e., u : [to,tf] — > Q, and 
u(-) piecewise continuous . 

Let u*(-) G U, let x* Q G T° and let x*(t) = (f)(t,t ,Xq,u*(-)) be the corresponding trajectory. 
Suppose that x*(tj) G T* , and suppose that («*(•), Xq) is optimal. Then there exist a number 



Pq > 0, and a function p* : [to, tf] — > i? n , not both identically zero, satisfying 

adjoint equation: p*(t) = -[^(t,x*(t),u*(t))]'p*(t), t <t<t f , (8.13) 

initial condition: p* (to) _LT° (xq) , (8.14) 

final condition: (p*(t f ) - p* Vip(x*(t f )))±T^ (x*{t f )) . (8.15) 
Furthermore, the maximum principle 

H(t,x*(t),u*(t),p*(t)) = M(t,x*(t),p*(t)) (8.16) 



holds for all t G [to, t/] except possibly for a finite set. [Here it) = p' f(t, x,u,), M(t,x,p) = 

sap{H(t, x,v,p)\v G O}]. 

Proof: We break the proof up into a series of steps. 

Step 1. By repeating the argument presented in the proof of Theorem 2 we can see that (8.15) is 
equivalent to 

p*(t f )'h < for all h G C(t f ) . (8.17) 

Step 2. Define two convex sets Si, S2 in R 1+m as follows: 

5 1 = {(y,h)\y>0, heTf(x*(t f ))}, 

5 2 = {(y,h)\y = ^ x {x*{t f ))K h G {C(t f ) + $(t f ,t Q )T°(x* )}} . 

We claim that the optimality of (u*(-),Xg) implies that Sifl Relative Interior (S2) = 4>- Suppose 
this is not the case. Then there exists h G T* (x*(tf)) such that 



Mx*{t f ))h>o , 



(8.18) 
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h G Interior{C(t/) + $(t/, t )T°(x%)} . (8.19) 

Now by assumption gl(x*(tf) has maximum rank. Since g[(x*(tf))h = it follows that the 
Implicit Function Theorem that for e > sufficiently small there exists h(e) G R n such that 

gf( x *( tf ) + h(e))=bf , (8.20) 

and, moreover, h(e) — ► 0, (l/e)/i(e) — ► /i as e — > 0. From (8.18) and Lemma 3 it follows that for 
e > sufficiently small there exists xq{e) G T° and u e (-) 6W such that 

a?*(t/) + /i(e) = 0{t f ,to,xo{e),u £ (-)) . 

Hence we can conclude from (8.20) that the pair (xo(e),u e (-)) satisfies the initial and final condi- 
tions, and the corresponding value of the objective function is 

^(x*(t f ) + h(e)) = V(s*(t/)) + M**(tf))He) + o(\h(e)\) , 
and since h(e) = eh + o(e) we get 

rp(x*(t f ) + h{e)) = *l){x*{t f )) + e)Mx*{t f ))h + o(e) ; 

but then from (8.18) 

il>(x*(t f ) + h(e))>,/>(x*(tf)) 

for e > sufficiently small, thereby contradicting the optimality of («*(■), Xq). 

Step 3. By the separation theorem for convex sets there exist fto G R,pi G -R n , not both zero, such 

that 

poy 1 + p'^ 1 > p y 2 + P> 2 for all (y\ h l ) G S x , i = 1, 2 . (8.21) 

Arguing in exactly the same fashion as in the proof of Lemma 1 of 7.2 we can conclude that (8.21) 
is equivalent to the following conditions: 



< &(t/,io) , (PoV^(x*(t / ))+p 1 )±r°(xS) , (8.23) 

and 

{pQ^ x {x*{t } )) +p' l )h < for all h G C%) . (8.24) 

If we letpo = Po andp*(t/) = p Vtp(x*(tf)) + pi then (8.22), (8.23), and (8.24) translate respec- 
tively into (8.15), (8.14), and (8.17). 
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8.2 Integral Objective Function 



In many control problems the objective function is not given as a function ip(x(tj)) of the final 
state, but rather as an integral of the form 

rt f 



f (t,x(t),u(t))dt . 



(8.25) 



The dynamics of the state, the boundary conditions, and control constraints are the same as before. 
We proceed to show how such objective functions can be treated as a special case of the problems 
of the last section. To this end we defined the augmented system with state variable x = (xq, x) G 
R 1+m as follows: 



x (t) 
x(t) 



f(t,x(t),u(t)) 



f (t,x(t),u(t)) 
f(t,x(t),u(t)) 



x 


= b° = 





g\x) 







The initial and final conditions which are of the form 

g°(x) = b°, g' (x) = V are augmented g°(x) 

and gf(x) = g*(x) = V . Evidently then the problem of maximizing (8.25) is equivalent to the 
problem of maximizing 

ip(x(tf)) = x {t f ) , 

subject to the augmented dynamics and constraints which is of the form treated in Theorem 3 of 

Section 1, and we get the following result. 

Theorem 1: Consider the optimal control problem (8.26): 

f (t,x(t),u{t))dt 



Maximize 



to 



(8.26) 



subject to 

dynamics: x(t) = f(t,x(t),u(t)), to <t<tf , 
initial conditions: g°(x(to)) = b° , 
final conditions: g* (x(t /)) = V , 
control constraint: u{-) £LU . 

Letu*(-) £ U, let x* Q G T° and let x*(t) = <j)(t,t ,x* ,u*(-)), and suppose that x*(t f ) £ T*. If 
(«*(■), Xq) is optimal, then there exists a function p* = (j)q,p*) : [to,tf] — >• R 1+m , not identically 
zero, and withp^t) = constant and p^(t) > 0, satisfying 

(augmented) adjoint equation: p* (t) = — [2j~(t, x*(t), u* (t))]' p* (t) , 
initial condition: p* (to) -LT° (xq) , 
final condition: p*(tf)±T^ (x*(tf)) . 
Fufhermore, the maximum principle 

H(t, x* (t) , p* (t) , u* (*)) = M(t, x* (t) , p* (t) ) 

holds for all t G [to>^/] except possibly for a finite set. [Here H(t, x,p,u) = p'f(t,x,u) = 
p f (t,x,u) +p'f(t,x,u), and M(t,x,p) = sup{H(t,x,p,v)\v G O}.] 

Finally, if /q and / do not explicitly depend on t, then M(t, x*(t),p*(t)) = constant. 

Exercise 1: Prove Theorem 1. (Hint: For the final part show that (d/dt) M(t, x*(t),p*(i}) = 0.) 
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8.3 Variable Final Time 

8.3.1 Main result. 

In the problem considered up to now the final time t f is assumed to be fixed. In many important 
cases the final time is itself a decision variable. One such case is the minimum-time problem where 
we want to transfer the state of the system from a given initial state to a specified final state in 
minimum time. More generally, consider the optimal control problem (8.27). 



ftf 

Maximize / fo(t,x(t),u(t))dt 
Jt 

subject to 

dynamics: x(t) = f{t,x, (t),u(t)), , t <t <t f , ^ 
initial condition: g°(x(to)) = b° , 
final condition: g* (x(t)f)) = W , 
control constraint: u(-) G U , 
final-time constraint: tf £ (to,oo) . 

We analyze (8.27) by converting the variable time interval [to, tf] into a fixed-time interval [0, 1]. 
This change of time-scale is achieved by regarding t as a new state variable and selecting a new 
time variable s which ranges over [0, 1]. The equation for t is 



ds 

with initial condition 



^ - a(s) , < s < 1 , 



t(0) = to • 

Here a(s) is a new control variable constrained by a(s) G (0, oo). Now if x(-) is the solution of 

x(t) = f{t, x{t),u(t)) ,t <t<t f , x(to) = x (8.28) 

and if we define 



z(s) = x(t(s)), v{s) = u(t(s)) , < s < 1 , 

then it is easy to see that z(-) is the solution of 

f s (s) = a(s)f(s,z(s),v(s)) , < s < 1 z(O) = x . (8.29) 

Conversely from the solution z(-) of (8.29) we can obtain the solution x(-) of (8.28) by 

x(t) = z(s(t)) , t <t<t f , 

where s(-) : [to>*/] — *• [0) 1] i s the functional inverse of s(t); in fact, s(-) is the solution of the 
differential equation s(t) = l/a(s(t)), s(to) = 0. 
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With these ideas in mind it is natural to consider the fixed-final-time optimal control problem 
(8.30), where the state vector (t, z) G R 1+m , and the control (a, v) G R 1+p : 

Maximize / fo(t(s), z(s), v (s))a(s)ds 
Jo 

subject to 

dynamics: (z(s),i(s)) = (f(t(s), z(s),v(s))a(s),a(s)), 
initial constraint: g°(z(0)) = b°, t(0) = to , 
final constraint: gf (z(l)) = V ,t(l) G R , 
control constraint: (v(s),a(s)) eOx (0, oo) 

for < s < 1 and v(-), a(-) piecewise continuous. 

The relation between problems (8.27) and (8.30) is established in the following result. 

Lemma 1: (i) Let x* Q G T°,u*(-) G U, t* f G (to, oo) and let x*(t) = (f)(t,to,XQ,u*(-)) be the 
coiTesponding trajectory. Suppose that x*(t*j) G T* , and suppose that (u*(-), Xqj*/) is optimal for 
(8.27). Define zg, «*(•), and a*(-) by 



u*(to + s(t* f - to)) , 0<s< 1 , 
- to) , < s < 1 . 

Then ((«*(•), «*(•))> 4) is optimal for (8.30). 

(ii) Let Zq G T , and let «*(•)) be an admissible control for (8.30) such that the correspond- 

ing trajectory (t* (■) , z* (■)) satisfies the final conditions of (8.30). Suppose that ((v* (■) , a* (■)) , Zq) 
is optimal for (8.30). Define Xq,u*(-) G U, and t* f by 

* _ * 

Xq — Z Q , 

u*(t) = v*(s*(i)) , t < t < t* f , 
t} =t*(l) , 

where s*(-) is functional inverse of t*(-). Then (u* (■) , Zq , t* f ) is optimal for (8.27). 
Exercise 1: Prove Lemma 1. 

Theorem 1: Let u*(-) G U, let x^ G T°, let t* f G (0, oo), and let 

x*(t) = <p(t,t ,XQ,u*(-)), t < t < tj, and suppose that x*(t*j) G T?. If (it*(-), Xq, t^) is optimal 
for (8.27), then there exists a function p* = (j>q,p*) ■ [to,tJ] — ► i? 1+m , not identically zero, and 
withpo(*) = constant andpo(*) > 0, satisfying 

(augmented) adjoint equation: 

P (t) = -[§(t,x*(t),u*(t))}>p*(t) , 



^0 = 

v*(s) = 
a*(s) = 



initial condition: p* (to) _LT° (x^) , 



(8.32) 
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final condition: p*(t* )±T^(x*(t* f )) . (8.33) 
Also the maximum principle 

H{t,x*{t),p*{t),u*{t)) = M{t,x*{t),p*{t)) , (8.34) 
holds for all t G [to, tf] except possibly for a finite set. Furthermore, t*j must be such that 

H(t},x*(t}),p*(t}),u*(t}))=0 . (8.35) 

Finally, if fo and / do not explicitly depend on t, then M(t, x*(t),p*(t)) = 0. 

Proof: By Lemma 1, z* = x* , v*(s) = u*(t + s(t* - t )) and a*(s) = (t* - t ) for < s < 1 
constitute an optimal solution for (8.30). The resulting trajectory is 

z*(s) = x*(t + s(t* f - t )),t*(s) = t + s(t* f - to), < s < 1 , so that in particular 
z*(l) = x*{t* f ). 

By Theorem 1 of Section 2, there exists a function A* = (Aq, A* , A* +1 ) : [0, 1] R 1+n+l , not 
identically zero, and with Ag(s) = constant and Ag(s) > 0, satisfying 



adjoint equation: 



' A5(t) - 




X*(t) 









{m(t*(s),z*(s),v*(s))]'X* (s) 

+m(t*(s),z*(s),v*(s))]>\*(s)}a*(s) 

{m(t*(s),z*(s),v*(s))]>\* (s) 

+ [%(t*(s),z*(s),v*(s))]>\*(s) }a*(s) 



(8.36) 



initial condition: X* (0) _LT° (zq ) 

final condition: \*(l)±T f (z*(l)) , \* n+1 (l) = . 
Furthermore, the maximum principle 

X* (s)fo(t*(s),z*(s),v*(s))a*(s) 

+X*( S yf(t*(s),z*(s),v*(s))a*(s) + X* n+1 (s)a*(s) 
= sup{[X* (s)f (t*(s),z*(s),w)f3 

+X*(syf(t*(s),z*(s),w)P + X* n+1 (s)/3]\wen, /3 € (0,oo)} 

holds for all s G [0, 1] except possibly for a finite set. 

Let s*{t) = {t- t )/{t* f - t ), t <t< t* f , and define p* = {p* ,p*) : [t ,t* f ] 

Po(t) = X*o(s*(t)), p*(t) = X*(s*(t)), t <t<t* f . 



(8.37) 
(8.38) 



(8.39) 

R 1+n by 

(8.40) 



First of all, p* is not identically zero. Because if p* = 0, then from (8.40) we have (Aq, A*) = and 
then from (8.36), A* +1 = constant, but from (8.38), A* +1 (l) = so that we would have A* = 
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which is a contradiction. It is trivial to verify that p*(-) satisfies (8.31), and, on the other hand (8.37) 
and (8.38) respectively imply (8.32) and (8.33). Next, (8.39) is equivalent to 



and 



\* (s)fo(t*(s),z*(s),v*(s)) 

+X*(syf(t*(s),z*(s),v*(s)) + X* n+I (s) = 



X* (s)fo(t*(s),z*(s),v*(s)) + X*(sYf(t*( S ),z*(s),v*(s)) 

= Sup {[X*(s)fo(t*(s),z*(s),w) + X*(s)'f(t*(s), z*(s), w)]\w G O}. 



(8.41) 



(8.42) 



Evidently (8.42) is equivalent to (8.34) and (8.35) follows from (8.41) and the fact that X* n+1 (l) = 0. 
Finally, the last assertion of the Theorem follows from (8.35) and the fact that M(t, x*(t),p*(t)) = 
constant if /o, / are not explicitly dependent on t. 

8.3.2 Minimum-time problems 

We consider the following special case of (8.27): 

'to 



Maximize / (-l)dt 
subject to 



dynamics: x(t) = f(t,x(t),u(t)), to<t<tf (8 43) 

initial condition: x(to) = xo , 
final condition: x(tf)=xj , 
control constraint: u(-) G U , 
final-time constraint: tf G (to,oo) . 

In (8.43), xo,Xf are fixed, so that the optimal control problem consists of finding a control which 
transfers the system from state xo at time to to state x f in minimum time. Applying Theorem 1 to 
this problem gives Theorem 2. 

Theorem 2: Let t*j G (to, oo) and let u* : [to, t*j] — > be optimal. Let x*(-) be the corresponding 
trajectory. Then there exists a function p* : [to,t*j] — ► R n , not identically zero, satisfying 

adjoint equation: p*(t) = —[^(t,x*(t),u*(t))]'p*(t), to < t < t*j , 
initial condition: p*{to) G R n , 
final condition: p*(t*j) G R n . 
Also the maximum principle 

H(t,x*(t),p*(t),u*(t)) = M(t,x*(t),p*(t)) (8.44) 

holds for all t G [to, tj] except possibly for a finite set. 
Finally, 

M{t* f ,x*{t f ),p*{t f ))>0 (8.45) 
and if / does not depend explicitly on t then 



M{t,x*{t),p*,{t)) = constant 



(8.46) 
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Exercise 2: Prove Theorem 2. 

We now study a simple example illustrating Theorem 2. Example 1: The motion of a particle is 
described by 

mx(t) + ax(t) = u(t) , 

where m = mass, a = coefficient of friction, u = applied force, and x = position of the particle. For 
simplicity we suppose that x G R, u G R and u{t) constrained by \u(t)\ < 1. Starting with an 
initial condition x(0) = xqi,x(0) = xq2 we wish to find an admissible control which brings the 
particle to the state x = 0, x = in minimum time. 



Solution: Taking x\ = x, x<i = x we rewrite the particle dynamics as 



xi(t) 
x 2 (t) 



1 

a 



x x {t) 
x 2 (t) 



+ 



u(t) , 



(8.47) 



where a = [a jm) > and b = (1/m) > 0. The control constraint set is 17 = [—1, 1]. 

Suppose that u*(-) is optimal and is the corresponding trajectory. By Theorem 2 there exists 
a non-zero solution p*(-) of 



" Pl(t) ' 




" " 




' Pl(t) ' 


_ m) . 




1 — a 




. Pt(t) _ 



(8.48) 



such that (8.44), (8.45), and (8.46) hold. Now the transition matrix function of the homogeneous 
part of (8.47) is 



*(t,r) 

so that the solution of (8.48) is 



1 I(i _ e -«(t-r)> 

e -a(t-r) 



" Pl(t) ' 




_ Plit) _ 





oi- 



and 



1 

I (1 _ e a t) e at 



pt(0) 

P2(0) 



P2W = ^i(0) + e at (-^(0)+p^(0)) 
The Hamiltonian H is given by 



(8.49) 



H(x*(t),p*(t),v) = (pl(t) - ap* 2 (t))x* 2 (t) + bp* 2 (t)v 

= e at (pt(0)-ap* 2 (0))x* 2 (t)+pb* 2 (t)v , 
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so that from the maximum principle we can immediately conclude that 

r +1 if P * 2 (t) > o, 

u*(t) = I -1 if p* 2 {t) < 0, (8.50) 
{ ? if p* 2 (t)=0 ■ 

Furthermore, since the right-hand side of (8.47) does not depend on t explicitly we must also have 

e at (pl(0) - ap* 2 (0))x* 2 (t) + bp* 2 {t)u*{t) = constant. (8.51) 

We now proceed to analyze the consequences of (8.49) and (8.50). First of all since p\(t) = 
Pi{0),p 2 (-) can have three qualitatively different forms. 

Case 1. -pl(0) + ap 2 (0) > 0: Evidently then, from (8.49) we see that p 2 (t) must be a strictly 
monotonically increasing function so that from (8.50) u*(-) can behave in one of two ways: 

either 



u*(t) 



-1 for t < i and p 2 (t) < for t < i, 
+1 for t > i and p 2 (t) > for t > i, 



or 



u*(t) = +1 and p* 2 {t) > for all t. 
Case 2. — p\(Q) + ap 2 {0) < : Evidently u*(-) can behave in one of two ways: 
either 



u*(t) 



or 



+1 for t < i and p 2 (t) > for t < i, 
-1 for t > i and p 2 (t) < for t > i, 



u*(t) = -1 and p*(t) < for all t. 



Case 3. -pi(0) + ap^O) = : In this case p 2 (t) = (l/a)p*(0). Also since p*(t) 0, we must 
have in this case p*(0) 7^ 0. Hence u*(-) we can behave in one of two ways: 



either 



u*(t) = +l and p%{t) = ±pl(0) > , 



or 

1 m *i 



u*(t) = -l and p* 2 (t) = ipl(0) < , 
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Thus, the optimal control u* is always equal to +1 or -1 and it can switch at most once between 
these two values. The optimal control is given by 

u*(t) = sgn p* 2 {t) 

= sgn[i^(0) + e- t (-^(0)+^(0))] . 

Thus the search for the optimal control reduces to finding p* (0) , p\ (0) such that the solution of the 
differential equation 

± = X2 ra 5?^ 

i 2 = - aX2 + 6sgn[^(0) + e Qi (-^(0)+^(0))] , K * J 1 

with initial condition 



cci(0) = xio,x 2 o = x 2 o (8.53) 

also satisfies the final condition 

x 1 (ty = 0, x 2 (t*) = , (8.54) 

for some t% > 0; and then t j is the minimum time. 

There are at least two ways of solving the two-point boundary value problem (8.52), (8.53), and 
(8.54). One way is to guess at the value of p* (0) and then integrate (8.52) and (8.53) forward in time 
and check if (8.54) is satisfied. If (8.54) is not satisfied then modify p*(0) and repeat. An alternative 
is to guess at the value of p*(0) and then integrate (8.52) and (8.54) backward in time and check of 
(8.53) is satisfied. The latter approach is more advantageous because we know that any trajectory 
obtained by this procedure is optimal for initial conditions which lie on the trajectory. Let us follow 
this procedure. 

Suppose we choose p*(0) such that — pl(0) = ap 2 (0) = and p 2 (0) > 0. Then we must have 
u*(t) = 1. Integrating (8.52) and (8.54) backward in time give us a trajectory £(t) where 

Ut) = -Ut) 

with 



6(0) - 6(0) = o . 

This gives 

6(t) = !(-i + ^),6(t) = !(l-e Qt ) , 

which is the curve OA in Figure 8.3. 

On the other hand, if p*(0) is such that -pl(0) + ap* 2 (ti) = and p* 2 (fi) < 0, then u*(t) = -1 
and we get 

6W = -U-t + ^) , 6(0 = "!(!- e at ) , 



which is the curve OB. 



.3. VARIABLE FINAL TIME 

u* = -1 
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Figure 8.3: Backward integration of (8.52) and (8.54). 



Next suppose p*(0) is such that -p*(0) + ap* 2 (0) > 0, and jp|(0) < 0. Then [(l/a^(0) + 
e at (— (l/a)p*(0) + P2(0))] will have a negative value for t G (0,t) and a positive value for t £ 
(£, cx)). Hence, if we integrate (8.52), (8.54) backwards in time we get trajectory £(t) where 



Ut) = a6(*)+ 



-b for t < t 
b for t>i 



with £i(0) = 0,6(0) = 0. This give us the curve OCD. Finally if p*(0) is such that -p*(0) + 
ap^O) < 0, and p^ ) < °> tnen n *(*) = 1 for i < £ and n*(t) = -1 for t > i, and we get the 
curve OEF. 

We see then that the optimal control u*(-) has the following characterizing properties: 



u*(t) 



1 if x*(t) is above BOA or on OA 
- 1 if x* (i) is below SOA or on OS . 



Hence we can synthesize the optimal control in feedback from: u*{t) = ip(x*(t)) where the 
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u* = 1 






u* = -1 


O 














— - V 

vr 



XI 



Figure 8.4: Optimal trajectories of Example 1. 
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function ip : R 2 ^{1,-1} is given by (see Figure 8.4) 

ip(x 1 ,x 2 ) = 



1 if (x\ , X2) is above BOA or on OA 
1 if (xi, x%) is below BOA or on OB . 



8.4 Linear System, Quadratic Cost 

An important class of problems which arise in practice is the case when the dynamics are linear and 
the objective function is quadratic. Specifically, consider the optimal control problem (8.55): 

r T 1 

Minimize / -[x'(t)P(t)x(t) + u'(t)Q(t)u(t)]dt 
Jo 2 

subject to 

dynamics: x(t) = A(t)x(t) + B(t)u(t), < t < T , (8.55) 
initial condition: x(0) = xq , 
final condition: G* x(t) = V , 

control constraint: u(t) E BP, u(-) piecewise continuous. 

In (8.56) we assume that P(t) is an n x n symmetric, positive semi-definite matrix whereas Q(t) is 
apxp symmetric, positive definite matrix. G* is a given £j x n matrix, and xq € R n , V G R £ f 
are given vectors. T is a fixed final time. 

We apply Theorem 1 of Section 2, so that we must search for a number Pq > and a function 
p* : [0, T] — > R n , not both zero, such that 

P*(t) = -p* (-P(t)x*(t)) - A'(t)p*(t) , (8.56) 

and 

p*(t)±T-f(x*(t)) = = 0} . (8.57) 

The Hamiltonian function is 

H(t,x*(t),p*(t),v) = -l P *[ x *(typ(t)x*(t)+v'Q(t)v] 

+ P *(t)'[A(t)x*(t) + B(t)v) 

so that the optimal control u*(t) must maximize 



2 

If p^ > 0, this will imply 



l plv'Q{t)v + p*{t)'B{t)v for v£RP . (8.58) 



u*(t) = ±Q-\t)B'(t)p*(t) , (8.59) 



whereas if p* Q = 0, then we must have 

p*(t)'B(t) = (8.60) 
because otherwise (8.58) cannot have a maximum. 
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We make the following assumption about the system dynamics. 
Assumption: The control system x(t) = A(t)x(t) + B(t)u(t) is controllable over the interval 
[0, T]. (See (Desoer [1970]) for a definition of controllability and for the properties we use below.) 
Let <5(i, r) be the transition matrix function of the homogeneous linear differential equation x(t) = 
A(t)x(t). Then the controllability assumption is equivalent to the statement that for any £ € R n 

f$(t, t)B{t) = , < r < T , implies £ = . (8.61) 

Next we claim that if the system is controllable then p$ ^ 0, because if p$ = then from (8.56) 
we can see that 

p*(t) = (<S>(T,t)y P *(T) 

and hence from (8.60) 

(p*(t)y$(T,t)B(t) = , < t < T , 

but then from (8.61) we get p*{T) = 0. Hence if p* = 0, then we must have p*(t) = which is a 
contradiction. Thus, under the controllability assumption, pg > 0, and hence the optimal control is 
given by (8.59). Now if p* Q > it is trivial that p*(t) = (1, (p* (t) /p* )) will satisfy all the necessary 
conditions so that we can assume that pj-J = 1. The optimal trajectory and the optimal control is 
obtained by solving the following two-point boundary value problem: 

x*(t) = A(t)x*(t) + B(t)Q- 1 (t)B'(t)p*(t) 

p(t)=P(t)x*(t)-A'(t)p*(t) 

x*(0) = x ,Gfx*(T) = bf , p*(T)±Tf(x*(T)) . 

For further details regarding the solution of this boundary value problem and for related topics see 
(See and Markus [1967]). 



8.5 The Singular Case 

In applying the necessary conditions derived in this chapter it sometimes happens that H (t, x* (t) , p* (t) , v) 
is independent of v for values of t lying in a non-zero interval. In such cases the maximum principle 
does not help in selecting the optimal value of the control. We are faced with the so-called singular 
case (because we are in trouble-not because the situation is rare). We illustrate this by analyzing 
Example 4 of Chapter 1 . 

The problem can be summarized as follows: 

Maximize / c(t)dt = [ (1 - s(t))f(k(t))dt 
Jo Jo 

subject to 

dynamics: k(t) = s(t)f(k(t)) - fj,k(t) , 0<t<T 
initial constraint: k(0) = ko , 
final constraint: k{t) £ R , 

control constraint: s(t) £ [0, 1], s(-) piecewise continuous. 
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We make the following assumptions regarding the production function /: 

fk(k)>0, fkk(K)<0 for all k , (8.62) 

lim f k (k) = oo . (8.63) 

Assumption (8.62) says that the marginal product of capital is positive and this marginal product 
decreases with increasing capital. Assumption (8.63) is mainly for technical convenience and can 
be dispensed with without difficulty. 

Now suppose that s* : [0,T] — ► [0, 1] is an optimal savings policy and let k*(t), < t < T, 
be the corresponding trajectory of the capital-to-labor ratio. Then by Theorem 1 of Section 2, there 
exist a number p$ > 0, and a function p* : [0, T] — > R, not both identically zero, such that 

p*(t) = -p* (l - s*(t))f k (k*(t)) - p*(t)[s*(t)f k (k*(t)) - /i] (8.64) 

with the final condition 



P*(T) = , (8.65) 

and the maximum principle holds. First of all, if p$ = then from (8.64) and (8.65) we must also 
have p*(t) = 0. Hence we must have Pq > and then by replacing (j>q,p*) by (1/Pq){Pq,P*) we 
can assume without losing generality that p$ = 1, so that (8.64) simplifies to 

p*(t) = -1(1 - s*(t))f k (k*(t)) - p*(t)[s*(t)f k (k*(t)) - fj] . (8.66) 

The maximum principle says that 

H(t,k*(t),p*(t),s) = (l-s)f(k*(t)) +p*(t)[sf(k*(t))-tik*(t)} 

is maximized over s G [0, 1] at s*(t), which immediately implies that 

r i if p*{t) > i 

s*(t) = < if p*(t) < 1 (8.67) 
[ ? if p*(t) = 1 

We analyze separately the three cases above. 
Case 1. p*(t) > 1, s*(t) = 1 : Then the dynamic equations become 



k*(t) = f(k*(t)) - iik*(t) , 
p*(t) = -p*(t)[f k (k*(t))-»] . 



(8.68) 



The behavior of the solutions of (8.68) is depicted in the (k,p) — , (k,t)— and (p,t)— planes in 
Figure 8.5. Here kc, kn are the solutions of f k (kc) — \i = and f{ku) — ^k = 0. Such solutions 
exist and are unique by virtue of the assumptions (8.62) and (8.63). Futhermore, we note from 

(8.62) that kc < ku, and f k {k) — fj, > according as k > kc whereas f{k) — fxk < according 

as k > ku- (See Figure 8.6.) 
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Figure 8.5: Illustration for Case 1. 



Case 2. p*(t) < l,s*(t) = 0: Then the dynamic equations are 

k*(t) = -fjtk*(t) , 

p*(t) = -f k (k*(t)) + w*(t) , 

giving rise to the behavior illustrated in Figure 8.7. 

Case 3. p*(t) = 1, s*(t) =?: (Possibly singular case.) Evidently if p*(t) = 1 only for a finite set of 
times t then we do not have to worry about this case. We face the singular case only if p* (t) = 1 
for t € I, where / is a non-zero interval. But then we have p*(t) = for t £ I so that from (8.66) 
we get 

-(1 - s*(t))f k (k*(t)) - [s*(t)f k (k*(t)) - fi] = for t € I , 
-fk(k*{t)) + fi = for t€l , 



or 



k*(t) = k G for t G I . (8.69) 



In turn then we must have k* (t) = for t G I so that 

8*(t)f(k G )- f iK G =0fort€l , 

and hence, 

ke 



**(*)= M/fe fOT teI ■ ( 8 - 7 °) 
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Figure 8.6: Illustration for assumptions (8.62), (8.63). 



Thus in the singular case the optimal solution is characterized by (8.69) and (8.70), as in Figure 8.8. 

We can now assemble separate cases to obtain the optimal control. First of all, from the final 
condition (8.65) we know that for t close to T, p*(t) < 1 so that we are in Case 2. We face two 
possibilities: Either (A) 



and then s*(t) = 0, k*(t) = k e~^, for < t < T, or (B) 

there exists t 2 G (0, T) such that p*(t 2 ) = 1 and p*(t) < 1 for t 2 < t < T . 

We then have three possibilities depending on the value of k*(t 2 ): 
(Bi) k*{t 2 ) < k G : then p*(t 2 ) < so that p*(t) > 1 for t < t 2 and we are in Case 1 so that 
s*(t) = 1 for t < t 2 . In particular we must have ko < k G . 

(Bii) k*(t 2 ) > k G : then p*( 2 ) > but then p*(t 2 + e) > 1 for e > sufficiently small and since 
p*(T) = there must exist £3 G (t 2) T) such that p*(ts) = 1. This contradicts the definition of t 2 
so that this possibility cannot arise. 

(Biii) k*(t 2 ) — ka ■ then we can have a singular arc in some interval (ti,t 2 ) so that p*(t) = 
l,k*(t) = ka, and s*(t) = fx{ka/f{kG)) for t G (h,t 2 ). For t < t\ we either have p*(t) > 
1, s*(t) > 1 if ko < k G , or we have p*(t) < 1, s*(t) = if k > k G . 
The various possibilities are illustrated in Figure 8.9. 

The capital-to-labor ratio k G is called the golden mean and the singular solution is called the 
golden path. The reason for this term is contained in the following exercise. 

Exercise 1: A capital-to-labor ratio k is said to be sustainable if there exists s G [0, 1] such that 
sf(k) — ixk = 0. Show that k G is the unique sustainable capital-to-labor ratio which maximizes 
sustainable consumption (1 — s)f(k). 



p*(t) < 1 for all t < [0,T] 
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Figure 8.7: Illustration for Case 2. 



8.6 Bibliographical Remarks 

The results presented in this chapter appeared in English in full detail for the first time in 1962 in the 
book by Pontryagin, et al. , cited earlier. That book contains many extensions and many examples 
and it is still an important source. However, the derivation of the maximum principle given in the 
book by Lee and Markus is more satisfactory. Several important generalizations of the maximum 
principle have appeared. On the one hand these include extensions to infinite-dimensional state 
spaces and on the other hand they allow for constraints on the state more general than merely initial 
and final constraints. For a unified, but mathematically difficult, treatment see (Neustadt [1969]). 
For a less rigorous treatment of state-space constraints see (Jacobson, et al, [1971]), whereas for a 
discussion of the singular case consult (Kelley, et al. [1968]). 

For an applications-oriented treatment of this subject the reader is referred to (Athans and Falb 
[1966]) and (Bryson and Ho [1969]). For applications of the maximum principle to optimal eco- 
nomic growth see (Shell [1967]). There is no single source of computational methods for optimal 
control problems. Among the many useful techniques which have been proposed see (Lasdon, et 
al, [1967]), (Kelley [1962]), (McReynolds [1966]), and (Balakrishnan and Neustadt [1964]); also 
consult (Jacobson and Mayne [1970]), and (Polak [1971]). 
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Figure 8.8: Case 3. The singular case. 
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t A ~ t 

Case [Biii) 



Figure 8.9: The optimal solution of example. 
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Chapter 9 

Dynamic programing 



SEQUENTIAL DECISION PROBLEMS: DYNAMIC PROGRAMMING FORMULATION 

The sequential decision problems discussed in the last three Chapters were analyzed by varia- 
tional methods, i.e., the necessary conditions for optimality were obtained by comparing the op- 
timal decision with decisions in a small neighborhood of the optimum. Dynamic programming 
(DP is a technique which compares the optimal decision with all the other decisions. This global 
comparison, therefore, leads to optimality conditions which are sufficient. The main advantage of 
DP, besides the fact that it give sufficiency conditions, is that DP permits very general problem for- 
mulations which do not require differentiability or convexity conditions or even the restriction to a 
finite-dimensional state space. The only disadvantage (which unfortunately often rules out its use) 
of DP is that it can easily give rise to enormous computational requirements. 

In the first section we develop the main recursion equation of DP for discrete-time problems. The 
second section deals with the continuous-time problem. Some general remarks and bibliographical 
references are collected in the final section. 

9.1 Discrete-time DP 

We consider a problem formulation similar to that of Chapter VI. However, for notational conve- 
nience we neglect final conditions and state-space constraints. 

JV-l 

Maximize ^ f (i,x(i),u(i)) + $(x(iV)) 
i=0 

subject to (9 1) 

dynamics: x(i + 1) = f(i, x(i),u(i)) , i = 0, 1, . . . , N — 1 , 
initial condition: x(0) = xq , 

control constraint: u(i) 6 fij , i = 0, 1, . . . , iV — 1 . 

In (9.1), the state x(i) and the control u(i) belong to arbitrary sets X and U respectively. X and U 
may be finite sets, or finite-dimensional vector spaces (as in the previous chapters), or even infinite- 
dimensional spaces, xq G X is fixed. The fi, are fixed subsets of U. Finally /o(i, •, •) : X x U —> 
R, <3? : X — ► R, f(i, •, •) : X x U — > X are fixed functions. 
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The main idea underlying DP involves embedding the optimal control problem (9.1), in which 
the system starts in state xq at time 0, into a family of optimal control problems with the same 
dynamics, objective function, and control constraint as in (9. 1) but with different initial states and 
initial times. More precisely, for each x G X and k between ) and N — 1, consider the following 
problem: 

N-l 

Maximize 23 f (i, x(i), u(i)) + <&{x{N)) , 

i=k 

subject to {92) 
dynamics: x(i + 1) = f(i, x(i),u(i)), i = k, k + 1, . . . , N — 1, 
initial condition: x(k) = x, 

control constraint: u{i) G fij, i = k, k + 1, ■, N — 1 . 

Since the initial time k and initial state x are the only parameters in the problem above, we will 
sometimes use the index (9.2)^. x to distinguish between different problems. We begin with an 
elementary but crucial observation. 

Lemma 1: Suppose u*(k), . . . ,u*(N — 1) is an optimal control for (9.2)^, and let x*(k) = 
x, x*(k + 1), . . . j x*(N) be the corresponding optimal trajectory. Then for any £, k < t < N — 
l,u*(£), . . . ,u*(N — 1) is an optimal control for (9.2)^ )X «m. 

Proof: Suppose not. Then there exists a control u(£),u(£ + 1), . . . , u(N — 1), with corresponding 
trajectory x(£) = x*(£),x(£ + 1), . . . , x(N), such that 



N-l 



£ /o(i, $(*),«(*)) + 
i=i 

N-l 

>Y t f {i,x*{i),u*®) + *(x*{N)) . 



(9.3) 



i=i 

But then consider the control u(k), . . . , u(N — 1) with 



«(*) 



u*(i) , i = k, ...,£— 1 
u{i) , i = £,..., N - 1 



and the corresponding trajectory, starting in state x at time k, is x(k), . . . , x(N) where 

x(i) = 

The value of the objective function corresponding to this control for the problem (9.2)^ :X is 



x*{i) , i = k,...,£ 

x(i) , i = £ + l,...,N . 



N-l 



23 fo(i,x(i),u(i)) + *(x(n)) 

i=k 

£-1 N-l 

= J3 /oft **(*)> «*(*)) + 2^ /o(*,x(i),«(i)) + *(a(JV)) 

i=fc i=£ 

> 23/o(i^*(0,«*(t)) + *(a;*(JV)) , 



i=k 
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by (9.3), so that u*(k), . . . , u*(N — 1) cannot be optimal for 9.2)^, x, contradicting the hypothesis, 
(end theorem) 

From now on we assume that an optimal solution to (9.2)fc jX exists for all < k < N — 1, and all 
iGl. Let V(k, x) be the maximum value of (9.2)^ >x . We call V the (maximum) value function. 
Theorem 1: Define V(N, •) by (V(N, x) = <3?(x). V(k, x) satisfies the backward recursion equa- 
tion 

V(k, x) = Max{/ , (k, x, u) + V(h,f(k, x, u, ))\u £ Q k }, < k < N - 1 . (9.4) 

Proof: Let x € X, let u*(k), . . . , u*(N — 1) be an optimal control for (9.2) k ^ x , and let x*(k) = 
x, . . . , x*(N) be the corresponding trajectory be x(k) = x, . . . , x(N). We have 

JV-l 

*£f Q (i,x*(i),u*(i))+$(x*(N)) 

rA (9.5) 

> ^f Q (i,x(i),u(i)) + <S>(x(N)) . 

i=k 

By Lemma 1 the left-hand side of (9.5) is equal to 

fo(k, x, u*(k)) + V(k + 1, f(k, x*,u*(k)) . 
On the other hand, by the definition of V we have 

N-l 

fo(i,x(i),u(i)) + $(x(iV)) = f (k,x,u(k)) 

i=k 
N 

HJ2 fo(i,x(i),u(i)) +^(x(N)) < f (k,x,u,(k)) +V(k + l,f(k,x,u(k))} , 

i=k+l 

with equality if and only if u(k+ 1), . . . , u(N — 1) is optimal for (9.2) fc+1 Combining these 

two facts we get 

fo(k, xu*(k)) + V(k + 1, f(k, x, u*(k))) 
> f (k, x, u{k)) + V(k + 1, f(x, k, u(k))) , 

for all u(k) E $7^, which is equivalent to (9.4). (end theorem) 

Corollary 1: Let u(k), . . . ,u(N — 1) be any control for the problem (9.2)^^ an d let x(k) = 
x, . . . , x(N) be the corresponding trajectory. Then 

V(£, x{t)) < f (£, x(£),u(£)) + V(£ + 1, f(£, x(£),u(£)), k < £ < N — 1 , 

and equality holds for all k < £ < N — 1 if and only if the control is optimal for (9.2)^. 
Corollary 2: For k = 0, 1, . . . , N — 1, let ip(k, •) : X — > be such that 

fo(k, x, iP(k, x)) + V(k + 1, f(k, x, i>(k, x)) 
= Max{f (k,x,u) + V(k + 1, f(k, x, u))\u G 14} • 

Then ip(k, •), k = 0, . . . , N — 1 is an optimal feedback control, i.e., for any fc, x the control 
. . . , u*(N - 1) defined by u*{£) = ip(£, x*(£)), k < £ < N - 1, where ' 
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x*(£+l) = f(£,x*(£),ijj(£,x*(£)), k < £ < N- 1 , x*{k) = x , 
is optimal for (a)k, x - 

Remark: Theorem 1 and Corollary 2 are the main results of DP. The recursion equation (9.4) al- 
lows us to compute the value function, and in evaluating the maximum in (9.4) we also obtain the 
optimum feedback control. Note that this feedback control is optimum for all initial conditions. 
However, unless we can find a "closed-form" analytic solution to (9.4), the DP formulation may 
necessitate a prohibitive amount of computation since we would have to compute and store the val- 
ues of V and ifi for all k and x. For instance, suppose n = 10 and the state-space X is a finite set 
with 20 elements. Then we have to compute and store 10 x 20 values of V, which is a reasonable 
amount. But now suppose X = R n and we approximate each dimension of x by 20 values. Then 
for N = 10, we have to compute and store 10x(20) n values of V. For n = 3 this number is 80,000, 
and for n = 5 it is 32,000,000, which is quite impractical for existing computers. This "curse of 
dimensionality" seriously limits the applicability of DP to problems where we cannot solve (9.4) 
analytically. 

• Exercise 1: An instructor is preparing to lead his class for a long hike. He assumes that each 
person can take up to W pounds in his knapsack. There are N possible items to choose from. 
Each unit of item i weighs W{ pounds. The instructor assigns a number U{ > for each 
unit of item i. These numbers represent the relative utility of that item during the hike. How 
many units of each item should be placed in each knapsack so as to maximize total utility? 
Formulate this problem by DP. 



9.2 Continuous-time DP 

We consider a continuous-time version of (9.2): 



Maximize $ f {t,x(t),u(t))dt + <$>(x(t f )) 
subject to 

dynamics: x(t) = f(t,x(t),u(t)) , to<t<tf (9.6) 
initial condition: x(0) = xq , 

control constraint: u: [to,tf]—>tt and tt(-) piecewise continuous. 

In (9.6), x £ R n , u G R p , O C R p . $ : R n — ► R is assumed differentiable and /q, / are assumed 
to satisfy the conditions stated in VIII. 1.1. 

As before, for to < t < tf and x G R n , let V(t, x) be the maximum value of the objective 
function over the interval [t,tf] starting in state x at time t. Then it is easy to see that V must satisfy 

rt+A 

V(t,x)=Max{ / (r,x(r),n(r))dr ^ 

+V(t + A, x(t + A))|u : [t, t + A] -» O}, A > , 

and 



V(t f ,x) = <S>(x) . (9.8) 
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In (9.7), x(t) is the solution of 

x(r) = /(t,x(t),u(t)) , t<T<t + A , 
x(t) = X . 

Let us suppose that V is differentiable in t and x. Then from (9.7) we get 

V(t, x) = Max{/ (t, x, it) A + V(t, x) + f£/(t, x, u)A 

+ff(i,x)A + o(A)|u G 0}, A > . 

Dividing by A > and letting A approach zero we get the Hamilton-Jacobi- Bellman partial 
differentiable equation for the value function: 

f^(i,x) + Max{f (t,x,u) + ^(t,x)f(t,x,u)\u G 0} = 0. (9.9) 

Theorem 1: Suppose there exists a differentiable function V : [to,tf] x R n — > R which satisfies 
(9.9) and the boundary condition (9.8). Suppose there exists a function ip : [to,tt] x R n — ► 17 
with if) piecewise continuous in t and Lipschitz in x, satisfying 

f (t,x,ijj(t,x)) + ^f(t,x,tp(t,x)) 

= M^{f (t,x,u) + ^f(t,x,u)\ u en} . K - ' 

Then ifi is an optimal feedback control for the problem (9.6), and V is the value function. 
Proof: Let t G [to?*/] an d x G -R n - Let u : [t, ij] — > O be any piecewise continuous control and 
let x(t) be the solution of 

x (t) = /(r, x(t), u(t)) , t<r <t f , ^ lV) 

x{t) = X . 

Let x* (r) be the solution of 

X*(t) = /(r,X*(T),^.(T,X*(T))) ,t<T<t f) 

x*(t) = X . 

Note that the hypothesis concerning ^ guarantees a solution of (9. 12). Let u* (r) = ^(r, x* , (r)) , t < 
t < tf. To show that ^ is an optimal feedback control we must show that 



^ f (tT,X*(T),U*(T))dT + <t>(x*(T)) 
f (T,X*(T),u(T))dT+$(x(tf)) . 



(9-13) 



To this end we note that 



<"tf 

V(t f ,x*(t f ))-V(t,x*(t))= I —( T ,x*(T))dr 



f 



u r dV , _ x . dV 



--€? {^(r,x*(r) + ^x*(r)}dr (9.14) 



F -0(T,x*(r),u*(r))d,T , 
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using (9.9), (9.10). On the other hand, 



(9.15) 



V(t f ,x(t f )) - V(t,x,(t)) = J\^( t ,x(t)) + ^ x (r)}dr 

<- J ' fo(r,x(T),u*(r))dT , 
using (9.9). From (9.14), (9.15), (9.8) and the fact that x*(t) = x(t) = x we conclude that 

V(t,x) = *(x*(t f )) + [ tf / (t,x*(t),«*(t)) 
ft 

> $(x(t f )) + y f (r,x(T),u(r))dT 

so that (9. 13) is proved. It also follows that V is the maximum value function. 

• Exercise 1: Obtain the value function and the optimal feedback control for the linear regula- 
tory problem: 



Minimize \x' (T)P{T)x(t) + \ J T Q {x'(t)P(t)x(t) 

+u'(i)Q(t)u(i)}dt 

subject to 

dynamics: x(t) = A(t)x(t) + B(t)u(t) , < t < T , 
initial condition: x(0) = xq , 
control constraint: u(t) £ RP , 

where P(t) = P'(t) is positive semi-definite, and Q(t) = Q'(t) is positive definite. [Hint: 
Obtain the partial differential equation satisfied by V(t, x) and try a solution of the form 
V(t, x) = x'R(t)x where R is unknown.] 



9.3 Miscellaneous Remarks 

There is vast literature dealing with the theory and applications of DP. The most elegant applications 
of DP are to various problems in operations research where one can obtain "closed-form" analytic 
solutions to be recursion equation for the value function. See (Bellman and Dreyfus [1952]) and 
(Wagner [1969]). In the case of sequential decision-making under uncertainties DP is about the 
only available general method. For an excellent introduction to this area of application see (Howard 
[I960]). For an important application of DP to computational considerations for optimal control 
problems see (Jacobson and Mayne [1970]). Larson [1968] has developed computational tech- 
niques which greatly increase the range of applicability of DP where closed-form solutions are not 
available. Finally, the book of Bellman [1957] is still excellent reading. [] 
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